Provisioning AMQP

Posted on

AMQP clients will allow you to declare your exchanges, queues, and bindings at the consumer level but that can cause problems as you use it more. You may get to the point that you will have to grep for all the declare methods in your code or run into problems trying to migrate to a new broker.

An alternative is to have consumers and producers take only the name of a queue or exchange and handle the rest outside of the application. This allows you to see and change in one place the configuration for all of your applications. When you need to provision a new broker it is done in a few seconds instead having to migrate some consumers, then all producers, then the rest of the consumers.

I've started writing and using Declare AMQP so that I can provision everything within Chef. It only supports the features I'm using but is very simple.

The migration is now much simpler as provisioning the server once is enough to make it ready for all applications. When I need to change exchanges or bindings I don't have to update any code. There is still the need to know which applications publish which routing key, but not a huge concern.

This has helped out as well configuring queues with specific priorities for the same type of tasks. Each application can be started with a queue to listen to and the configuration for both the broker and applications remains in one place.


Prioritizing Emails with RabbitMQ

Posted on

After you move a few tasks to the background with RabbitMQ you may realize that you eventually need to support different priorities for the same type of tasks, sending bulk email after you send transactional email. RabbitMQ doesn't have priorities so you wind up having to use separate queues for each priority.

You should already have a worker that can send the email, just now you need setup RabbitMQ with priorities.

The main exchange you use, email, should be declared either topic or direct and will take all of the messages you intend to send but when declared you should include an alternate exchange of email-undeliverable that is declared as a fanout exchange. Now you just need a default queue bound to the default routing key for the email exchange and also bound to email-undeliverable. Now every email your try to send that doesn't have a specifically prioritized queue will be routed to the default queue.

All you need now is to start your workers consuming from each queue you create.