huey, a little task queue¶
a lightweight alternative, huey is:
- written in python (2.7+, 3.4+)
- only dependency is the Python Redis client
- clean and simple APIs
huey supports:
- multi-process, multi-thread or greenlet task execution models
- schedule tasks to execute at a given time, or after a given delay
- schedule recurring tasks, like a crontab
- automatically retry tasks that fail
- task result storage
- task locking
- task pipelines and chains
- consumer publishes event stream, allowing high-fidelity monitoring
At a glance¶
Use the task() and periodic_task() decorators to
turn functions into tasks that will be run by the consumer:
from huey import RedisHuey, crontab
huey = RedisHuey('my-app', host='redis.myapp.com')
@huey.task()
def add_numbers(a, b):
return a + b
@huey.periodic_task(crontab(minute='0', hour='3'))
def nightly_backup():
sync_all_data()
Here’s how to run the consumer with four worker processes (good setup for CPU-intensive processing):
$ huey_consumer.py my_app.huey -k process -w 4
If your work-loads are mostly IO-bound, you can run the consumer with threads or greenlets instead. Because greenlets are so lightweight, you can run quite a few of them efficiently:
$ huey_consumer.py my_app.huey -k greenlet -w 32
Redis¶
Huey’s design and feature-set are, to a large extent, informed by the capabilities of the Redis database. Redis is a fantastic fit for a lightweight task queueing library like Huey: it’s self-contained, versatile, and can be a multi-purpose solution for other web-application tasks like caching, event publishing, analytics, rate-limiting, and more.
Although Huey was designed with Redis in mind, the storage system implements a simple API and many other tools could be used instead of Redis if that’s your preference. Huey ships with an alternative storage implementation that uses SQLite Storage.
Table of contents¶
Huey is named in honor of my cat
