Using RabbitMQ Beyond Queueing

UPDATED 2008-11-12: Adjusted Failover section below (additions in italic) based on a thread on rabbit-discuss.

I am a big fan of RabbitMQ, an implementation of Advanced Message Queueing Protocol. In this post I am going to provide an overview how RabbitMQ can be used beyond simple queueing and pubsub. For more background on this topic, please see a list of messaging scenarios that RabbitMQ supports.

Queueing

Broker can take a message from producer and keep it until a consumer shows up. To survive broker restarts, queues in this case should be durable, with auto-delete set to false, and messages should be published with delivery mode of 2 (which means persistent). This pattern can also be helpful if consumer is temporarily unable to keep up with incoming message flow - queueing allows producer and consumer to keep going at their own pace, and will make sure all messages are consumed eventually.

Multiplex

This is when multiple producers publish messages to be routed to a single consumer. You can either have all your producers publish to the same fanout exchange, or to the same direct exchange with the same routing key, or to a topic exchange with a routing key that matches that of consumer. The latter case allows producers to publish “flights.aa.ord” and “flights.ua.sfo”, while the consumer can be reading all of these with “flights.#” (* matches a single word, # matches zero or more words).

Demultiplex

This is when a single producer publishes messages that are routed to multiple different consumers. It can be a topic exchange, producer could publish “order.book” and “order.cd”, while orders for books and CDs are handled by different systems.

Instant Feedback (Queueing Bypass)

Producer can instruct broker not to queue a message at all and return it to sender if a consumer is not currently available to read it (this is achieved by setting immediate flag to true in basic.publish method). Can be helpful in scenarios where message content is time sensitive - it needs to be processed now or an error must be returned.

Duplicating

In AMQP, a message is delivered to all queues bound to a given exchange, if a queue meets routing criteria (for different types of exchanges, these criteria are different). For example, a message published with “prod.server01.disk.full” key can be simultaneously routed to “prod.#” queue (for production logger to keep track of all events in production environment) and “#.disk.full” queue (for an archiver process that removes old logs). Very powerful feature, and it works with direct and fanout exchanges as well.

Load balancing

If multiple consumers read from the same queue, RabbitMQ broker will automatically load balance messages between all available consumers. Each message will be sent to one consumer at a time.

Failover

In no_ack=false mode, a consumer must eventually explicitly acknowledge receipt of each message, individually or as a group (this does not mean that a message must be ack'ed before next one can be received). If a consumer disconnects without acknowledging, unack'ed messages are automatically re-queued for another consumer. This helps achieve consumer failover in response to crashes or loss of network connectivity.

Relaying

If producers and consumers do not have direct line of sight network-wise (for example, they are behind NAT or are located on private subnets), RabbitMQ can provide the connectivity by serving as a message relayer. Both producers and consumers must be able to establish client connections to broker (AMQP official port is TCP 5672) and then they can exchange messages.

Conclusion

Some of these patterns can be mixed and matched, which further expands a set of problems where RabbitMQ can help you achieve a distributed messaging nirvana.

Categories: rabbitmq |