huey, a little task queue

http://media.charlesleifer.com/blog/photos/huey-logo.png

a lightweight alternative, huey is:

  • written in python
  • only dependency is the Python Redis client
  • clean and simple APIs

huey supports:

  • multi-process, multi-thread or greenlet task execution models
  • schedule tasks to execute at a given time, or after a given delay
  • schedule recurring tasks, like a crontab
  • automatically retry tasks that fail
  • task result storage
  • consumer publishes event stream, allowing high-fidelity monitoring
http://i.imgur.com/2EpRs.jpg

At a glance

Use the task() and periodic_task() decorators to turn functions into tasks that will be run by the consumer:

from huey import RedisHuey, crontab

huey = RedisHuey('my-app', host='redis.myapp.com')

@huey.task()
def add_numbers(a, b):
    return a + b

@huey.periodic_task(crontab(minute='0', hour='3'))
def nightly_backup():
    sync_all_data()

Here’s how to run the consumer with four worker processes (good setup for CPU-intensive processing):

$ huey_consumer.py my_app.huey -k process -w 4

If your work-loads are mostly IO-bound, you can run the consumer with threads or greenlets instead. Because greenlets are so lightweight, you can run quite a few of them efficiently:

$ huey_consumer.py my_app.huey -k greenlet -w 32

Redis

Huey’s design and feature-set are, to a large extent, informed by the capabilities of the Redis database. Redis is a fantastic fit for a lightweight task queueing library like Huey: it’s self-contained, versatile, and can be a multi-purpose solution for other web-application tasks like caching, event publishing, analytics, rate-limiting, and more.

Although Huey was designed with Redis in mind, the storage system implements a simple API and many other tools could be used instead of Redis if that’s your preference. Huey ships with an alternative storage implementation that uses SQLite Storage.

Indices and tables