In the true spirit of open source community that has formed around RabbitMQ in the past several years and continues growing every week, on the mailing list we have recently tackled an issue when one runs “rabbitmqctl status” and gets “badrpc,nodedown” response, while broker is running as evidenced by ps output. Check out a thread on “broker runs; cant’ get status” here. The issue is centered around erlang’s security and communications mechanism in distributed mode. Here is a tentative step-by-step that can help you resolve the issue.

Are you running broker as user rabbitmq and rabbitmqctl as root or rabbitmq? If not, please stop and fix this. This is not a requirement per se, but this represents canonical installation of rabbitmq broker. You can certainly hack your scripts to work around this requirement, but you are on your own if you do.

Double check that broker is in fact running. Use ps, netstat -lptn (look for port 5672 unless you overrode it in /etc/default/rabbitmq). Telnet to localhost on port 5672, type AMQP and press ENTER several times. You should get response that at least will show AMQ. Check logs at /var/log/rabbitmq to verify that broker saw your connection attempt.

Next step is to start "erl -sname foo -cookie coo" in shell and run this command: "net_adm:names()."

If this command returns ok followed by a list of nodes within 1 or 2 seconds, check if rabbit is there. If it is, it’s very likely that you have users mixed up above. Please double check. If rabbit node is not listed, double check that rabbit broker is still running.

If this command returns {error,address}, there is a problem with your instance of EPMD, erlang naming daemon (man net_adm). First, check if it’s running (it most likely is running). Then, in erlang, run ”net_adm:localhost().” Exit erlang, and try to connect to exactly the name you got from net_adm:localhost() on port 4369 (epmd). This shouldn’t work and should timeout. If it doesn’t time out, you shouldn’t have gotten {error,address}.

The problem most likely will be that name as returned by net_adm:localhost() is associated with some IP in /etc/hosts or in DNS which is either not accessible from this host, or firewalled off. An entry in /etc/hosts that associates this name with 127.0.0.1 or one of other IPs on this server should fix the problem.

Alternatively, net_adm:names() may time out with {error, timeout}. We have seen it caused by snoopy in the past. Remove snoopy or do not install it system wide in /etc/ld.so.preload, and you should be fine.

If these steps did not help, please leave a comment below or join rabbitmq-discuss and ask your question there, and we’ll help!

UPDATE 2009-04-17: Your host's name as shown by net_adm:localhost() does not technically need to be defined in /etc/hosts. But if it's not there, it should not be defined anywhere - when you do "ping name", you should get "ping: unknown host name". I have seen at least one case when it worked this way. This is somewhat unverified though.

I recently glanced over Apache httpd logs for this blog. Among other things, I discovered several bots that were making quite a few useless requests, thus driving load on the machine. It wasn’t a big deal but a matter of principle. If all of us webmasters start paying attention to misbehaving bots and block them, their authors or maintainers might finally learn how to play by the rules - it’s not difficult really.

Let’s say you are running a blog and hosting your own feed. Regular crawler bots from search engines like Google, Yahoo, MSN, etc will always check robots.txt before sending next batch of requests, and they won’t ask for the same URL very frequently.

The second type of bots you are going to deal with, are RSS reader bots. There is no reason for them to hit anything but URL for your feeds. Such bots usually will not check robots.txt. Their user agent string (which you can see if you enable CustomLog /path/to/log combined) will usually have “N subscribers,” which can give you a rough idea how many people subscribe to your feed via this service. These bots may hit you with varying regularity, ranging from half an hour to several hours between requests. Additionally, well behaving RSS reader bots will always include If-Modified-Since in their requests - you will see in your logs that your usual response to these queries is 304 Not Modified, and only once after new post is published you should see 200 OK response.

So what did I see in my logs? First, I noticed a bot requesting / from 2 different IPs every hour, without including If-Modified-Since. Wasteful and negligent. If this bot does not know how to appreciate my server resources, I am sure it won’t miss my content - block!

<Location />
order allow,deny
allow from all
# BadBot1 is from its User-Agent string
SetEnvIf User-Agent BadBot1 DenyBot=1
deny from env=DenyBot
</Location>

Then I noticed several RSS reader bots that were requesting /feed/ way too frequently (every 10 minutes for 1 subscriber and every 60 minutes for one susbcriber) and were very inconsistent with If-Modified-Since - I couldn’t detect logic in their requests, but sometimes I saw 304 and sometimes I saw 200 response, even when there was no new content on my site.

I didn’t feel right blocking these altogether, so instead what I did is I opened a particular 1 hour window during the night when I allow such bots to get the feed’s content - all other times, their requests are blocked.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} .*BadBot2.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*BadBot3.*
RewriteCond %{REQUEST_URI} /feed/
RewriteCond /etc/allow_bad_bots !-f
RewriteRule . - [forbidden]

Voila! If file /etc/allow_bad_bots exists (I create and delete it from cron, it exists on my system between 1am and 2am), requests from these bots will succeed. During the rest of the day, these rude bots are getting 403 Forbidden.

I recently came across a fascinating book - Pop! Why Bubbles Are Great For The Economy, by Daniel Gross, author and Moneybox columnist for Slate. In it, he focuses on positive consequences of several bubbles in US economic history (namely telegraph, railroads, financial New Deal, Internet, and real state) and shows how “the frenzy of irrational economic enthusiasm lays the groundwork for sober-minded opportunities, growth and innovation.” (All quotes in this post are from the book.) In other words, excesses left when a bubble bursts, are later successfully re-used as a foundation for new businesses, and even more importantly - without those excesses, new businesses could not get so successful so fast.

The first chapter of the book titled Bubbles "R" Us talks about bubbles in general. Let me do a short walkthrough.

Bubbles "R" Us

What's a [...] bubble? [...] In every generation, people arise to proclaim that a new technology or a new set of economic assumptions and financial tools promise untold riches. The new, new thing will both alter history and free us from its musty grasp. As a result, the old rules simply no longer apply. [...] Evangelists and proselytizers urge people to abandon reason and steady habits, and browbeat those who steadfastly stick to their sense of rationality. [...] As the bubble takes shape, a chorus of pundits, shills, and true believers concoct eminently solid reasons why this time is different. [...] As the bubble takes flight, the sector swells [...]. Now the phenomenon crosses over into the larger consciousness [...]. The bubble fully aloft, investors and managers are seized by pervasive me-tooism, a conviction that the fourth, fifth and sixth movers will enjoy first-mover advantage. [...] Meanwhile, an important parallel process takes place. During bubbles, a great deal of money and energy is spent building up the mental infrastructure surrounding a new technology - convincing people to invest, to use new media, to make Internet phone calls, to reserve hotel rooms by telegraph, to send grain by rails instead of by canal barge, to buy stocks and investment trusts instead of leaving cash under the mattress. [...] Next comes the end stage. [...] Finally, when economic reality catches up to dreams and hype, the bubble bursts. [...]

Disruptive Technologies As Bubbles

As I was reading the book, I kept thinking that each major technology shift in IT essentially behaves like a bubble. No one wants to use it at first, or rewrite their applications for something that was just released. But as times goes on, there are early adopters. And then there are people and sometimes organizations who throw their weight behind a newcomer and distribute the knowledge. And then, like a bubble, it reaches a wider audience and crosses over to new domains and fields.

Unlike an economic bubble however, there is usually no or little excess of internal build out, prices for raw materials are rarely significantly impacted and me-tooism is generally not a problem (the more vendors, the better). And most importantly, unlike economic bubbles, new technologies or computing paradigms don’t have to burst under pressure of unrealistic assumptions - they can continue growing, mature and then peacefully and gradually give way to something new and even more exciting.

And as economic bubbles leave something fundamentally positive behind, so do the new technologies.

I strongly believe that a single most prominent accomplishment of cloud computing is widespread use of API to control one's hardware infrastructure (Infrastructure as a Service). Look at current IaaS landscape - Amazon EC2, ElasticHosts, Flexiscale, GoGrid and probably many more. All of them offer rich end-user APIs to do things that used to require an on-site team and certain amount of physical labor. And now it all has been automated.

Also, check out this Cloud Taxonomy and Ontology diagram that Christopher Hoff is working on. Note how API block appears in the center and even has its own color to emphasize its importance!

Hardware infrastructure manageable from software is too awesome to ignore - are you in the cloud yet?

A quick note. This blog is getting a lot of google referrals for people looking for “rabbitmq rails”. I just wanted to say that I do not have good up-to-date material on the subject. Instead, check out this thread on ruby-amqp mailing list and consider tmm1-amqp gem for your project.

This is a semi-serious, semi-joking post.

Earlier this month, Seth Godin, one of the leading thinkers of our digital age, wrote a post titled Do ads work?

If the local bank were offering a sale on dollar bills, ninety cents each, how many would you buy? Most rational people would say, "I'll take them all please." Especially if you had thirty days to pay for them. So, why, precisely, do you have an ad budget? If your ads work, if you can measure them and they return more profit than they cost, why not keep buying them until they stop working? And if they don't work, why are you running them

Some of you may recall my love towards parallels between IT and other disciplines. If we twist these words a bit, we get this:

So, why, precisely do you have specific maintenance windows which are the only times when changes in production are allowed - don't they just unnecessarily delay the changes that inevitably are going to be made? If the changes you are about to make work, if you can measure them and they improve your product more than they cost, why not keep making these changes until they stop working? And if they don't work, why are you making these changes?

I know it’s a stretch but worth thinking about nevertheless… Release often FTW.