Worker Process

Posted on

Part of a current project at work involves writing many stand-alone processes to handle events from RabbitMQ. The worker processes are managed by Supervisor but should gracefully handle SIGTERM and generally follow a common pattern.

The main design is splitting out how worker's should run from what they are supposed to be doing. Extending the BaseWorker class will give a main classmethod used to start new workers.

The BaseWorker class is split from the WorkerRunner class giving a common interface for all workers and Unixy interaction.

You can take a look at what I have so far on the projects page.


Packaging

Posted on

Roughly 2 weeks ago I started Plug which aims to create a package format for Python daemons. The project started after seeing how Supervisor handles 150+ processes.

A current project at work can easily have many daemon processes with differing number of running instances that may need to be adjusted frequently. Deploying with Supervisor can be a problem given the amount of time Supervisor would take to start/stop processes.

Plug installs each package into a virtualenv then uses runit to manage each daemon instance.

I have a prototype version working now with a packaged version to come in the next few weeks after giving it more testing.

The biggest issue is after watching To Package or Not to Package I am falling in more of the "to package" crowd and despite Plug being a packaging solution smells a bit too much of NIH.