Hacking RabbitMQ - Remote Queue Forwarders
Since earlier this year when I got my copy of Erlang book, I’ve wanted to do something unconventional with RabbitMQ source. I finally came up with an idea, which is somewhat interesting and maybe even useful, and could be done by an Erlang beginner like myself.
Some background first. Each Erlang program in general consists of multiple Erlang processes that send and receive messages to/from each other. These are not your regular processes - these processes are running inside the Erlang VM, and are not mapped to either processes or threads on the system level in any way. They are cheap to create, and communications between them are fast. Each process has a process ID (pid) associated with it. If multiple Erlang VMs share a cookie (a piece of text which is used as a security token), processes on any Erlang VM can freely talk to each other without any code modification.
RabbitMQ implements each queue as an Erlang process. You can think of AMQP exchanges as pieces of routing logic - when a message arrives, Rabbit applies the routing rules to its routing key and sends the message to all eligible queues. Since each queue is a process and has a pid, in a nutshell exchanges send the message to several pids.
In non-clustered mode, Rabbit routing function will return a list of local PIDs by selecting values stored in a mnesia table (rabbit_durable_queue and rabbit_queue). But since RabbitMQ can also run in clustered mode, developers already implemented a way to send messages to queues on remote nodes. So when I hacked rabbit_durable_queue and rabbit_queue tables and replaced PIDs pointing to local queues with PIDs pointing to remote queues, I got myself a remote queue forwarder.
How is this useful you might ask. RabbitMQ supports remote queues only in clustered mode. The way clustering is implemented today (using mnesia in distributed mode), it’s not recommended to run a rabbit cluster over non-LAN links. This is because if 2 rabbit nodes lose and then regain connectivity to each other, the cluster may enter a “partitioned network” state, which effectively means that rabbit cluster is not functional (in other words, mnesia sacrifices tolerance to partitioning in order to achieve consistency and availability - recall CAP theorem). With remote queue forwarding, you don’t need to set up clustering and hence “partitioned network” state won’t affect you by design - and that’s what mattered to me.
On the other hand, remote queue forwarding can potentially break some AMQP guarantees. For example, if a remote node is temporarily unavailable, Rabbit won’t queue the message for later re-delivery (because such situation is currently impossible in unhacked Rabbit). It means that YOU SHOULD NOT USE THIS HACK UNLESS YOU KNOW WHAT YOU ARE DOING.
The hack works for me though in the following scenario. I publish messages with immediate=true (indicates that payload is time sensitive). Messages are sent to N local queues. These N local queues are forwarded to N remote nodes, with a consumer attached to 127.0.0.1 on each node. In this project, I don’t rely on any AMQP guarantees - I discard basic.return commands and can tolerate occasional message not reaching some or even all consumers.
I doubt anyone will find this hack in its current form useful, but just in case I uploaded it to http://github.com/somic/rabbit_queue_forwarder.
PS. After I started this small project, Tony Garnock-Jones announced that his work on pluggable exchange types has been added to default branch. I still went ahead and published my post, but please note that pluggable exchange types could probably achieve similar effect more cleanly.