Graphite RabbitMQ Integration

I started a new project on github - http://github.com/somic/graphite-rabbitmq. It currently includes a couple of tools written in Python which facilitate sending data to Graphite via RabbitMQ instead of connecting directly to the service using TCP.

Graphite CLI Screenshot

Graphite is a flexible and powerful tool to build charts. It’s also a data series analytics framework. It was developed inside Orbitz by my former colleague, originally for use within a single group (of which I was a part). However, its power did not remain a secret for too long - it quickly spread to entire organization and became an irreplaceable tool for both development and engineering/operations. Graphite was then open-sourced under Apache license. It currently lives at http://graphite.wikidot.com/

The key to Graphite’s power, in addition to dynamic web UI, an improved RRD implementation called “whisper” (read this FAQ - highly recommend!) web-based command line with auto-completion which allows you to overlay any metrics on a single chart, IMHO is the fact that you are in control what kind of data to send to it, how often, and how to set up hierarchies of your metrics - by environment, by machine type, by datacenter, etc. Graphite doesn’t do its own polling that won’t scale to hundreds or thousands of metrics. Nor does it enforce anything but the fact that your metrics are dot-separated hierarchies (as in routing keys of AMQP topic exchanges - my.metric.name) and that their values are numeric (int or float).

If you are still reading this but still are not convinced that it’s the way to go, I’ve got one last argument. If you already use RabbitMQ to publish and consume data, wouldn’t it be nice to get a powerful charts without touching your application AND without installing agents on your publishers or consumers? Recall the duplication pattern of RabbitMQ - you can fork the incoming stream of messages into another queue (without impacting your original consumers and the queues to which they attach) and set up Graphite+RabbitMQ off of this new queue.

If you are planning to run multiple carbon instances, remember that heavy lifting (writing to disk) is actually performed by another process called carbon-persister.py (it’s started by carbon-agent, with communications over a pipe) - try to avoid multiple persisters writing data within the same hierarchy to avoid slow down and possible data corruption. RabbitMQ can help you sort out what messages go where, thus minimizing this risk.

I am very excited about future opportunities that a Graphite-RabbitMQ combination can deliver, and I hope someone finds my scripts useful. Both tools bring a lot of awesomeness to the table, and nicely complement each other forming a great charts and data series analytics solution you have been searching for. Check it out!

Categories: rabbitmq | python |

Comments (4)

John Apps // 21 May 2009

Dmitriy, great article, good topic, and very much in the spirit of AMQP! Have you, by any chance, managed to glance a www.ZeroMQ.org (0MQ) yet? Well worth a look and a messaging system which, I believe, would very much lend itself to your work here.

Cheers, John

Dmitriy // 21 May 2009

Thanks John.

I know about 0MQ but haven't had a chance to check it out in detail yet.

I hope that graphite + rabbitmq integration scripts will work against all (or at least most) AMQP brokers. Interop FTW as they say :)

- Dmitriy

Paul // 27 May 2009

Wicked work - you couldn't have announced this at a better time. I've just managed to setup a dashboard of sorts using graphite plus this bit of code, plugging it into my rabbitMQ data feed - all in about 10 minutes. It does what it says on the tin and straight out of the box. Well done!

Dmitriy // 27 May 2009

Paul, this is awesome. I am very happy to hear it worked well for you.