I've been interested in simplifying lately.... using Chef (solo) to configure boxes, switching to Postgres from MySQL to get decent text search without having to run sphinx, etc. My goal recently has been to cut down the number of moving parts in my deployments to make my life a little easier. So when a client project required background processing, I re-evaluated my stock approach to see if I could simplify a bit.
I've been doing background stuff with Rails for a long time, so I've used backgroundrb, backgroundjob, delayed_job, and, most recently, resque. I've enjoyed them all. :) This time around, I decided to give delayed_job another go in order to avoid having to add redis to the deployment mix just for background jobs. There won't be a lot of background jobs in this project, so I don't need a number of workers (which would be perfect for resque), so delayed_job seemed a good fit to run these jobs.
The problem was that I still needed a daemon running (the worker process) to get jobs to be processed, and I was looking for a way to avoid running (and monitoring) yet another daemon. Enter pedro's fork of delayed_job, which automagically spins up and spins down a worker for you as jobs get created and the queue gets cleared. Since my partner on this project went with collectiveidea's fork of delayed job, I decided to combine pedro's work with collectiveidea's more recent updates into my own fork of collectiveidea's 2.0 branch (which is targeted at Rails 2.x).
Using this fork of delayed_job, you don't need to worry about running a worker process. When a job is created, a check is made to see if a worker is running, and, if not, one is spawned (either as a process via the rush gem if you're on a standard box/slice, or as a heroku worker via the heroku gem if you're on heroku). When the job queue gets processed, the worker is stopped if there are no more jobs in the queue. It is so awesome. :)
Since we are deploying this client's code to both heroku (for staging) and EY cloud (for production), the delayed_job initializer looks like this:
Auto-scaling delayed_job workers makes everybody happy — long-running tasks go in the background, and you don't have to add even so much as a chef recipe to your deployment. :)
Comments