docs: add Task Queues category intro

Task Queues had no intro, so its meta description fell back to generic text and readers got no guidance on which library to pick.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Vinta Chen
2026-09-27 08:03:28 +08:00
co-authored by Claude
parent da7c0b7342
commit 4e020707e7
@@ -0,0 +1,22 @@
Two questions pick a Python task queue: is your app async, and what's your broker? Taskiq is for asyncio. Celery takes RabbitMQ or Redis, RQ Redis, Huey SQLite.
How to choose:
- An asyncio app, like FastAPI: Taskiq
- A batteries-included queue on RabbitMQ, Redis, or Amazon SQS, or tasks sent from non-Python code: Celery
- The simplest job queue, on Redis or Valkey: RQ
- No broker server to run, with the queue in SQLite or the Postgres you already have: Huey
- A production backend for Django's task framework: Huey
- Retries with backoff and acks after processing by default, on RabbitMQ or Redis: Dramatiq
Taskiq's docs call it [an asyncio Celery implementation](https://taskiq-python.github.io/): you mark tasks with `@broker.task` and send them with `await task.kiq()`. Since sending is async only, its docs [point fully synchronous projects to Celery or Dramatiq](https://taskiq-python.github.io/guide/). For production, its docs [highly recommend](https://taskiq-python.github.io/guide/getting-started.html) taskiq-aio-pika or taskiq-nats as the broker and taskiq-redis as the result backend. In a FastAPI app, [taskiq-fastapi](https://taskiq-python.github.io/framework_integrations/taskiq-with-fastapi.html) connects the broker to your app.
Celery is a task queue [with batteries included](https://docs.celeryq.dev/en/stable/getting-started/first-steps-with-celery.html): [celery beat](https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html) runs periodic tasks, and [Django support](https://docs.celeryq.dev/en/stable/django/first-steps-with-django.html) comes out of the box. It runs on RabbitMQ, Redis, or Amazon SQS. Its docs call RabbitMQ [an excellent choice for production](https://docs.celeryq.dev/en/stable/getting-started/first-steps-with-celery.html#choosing-a-broker) and warn that Redis is more likely to lose data on a sudden shutdown or power failure. Its protocol has clients in other languages, so non-Python code can send tasks too. Celery [doesn't support Windows](https://docs.celeryq.dev/en/stable/getting-started/introduction.html), though.
RQ is a simple job queue on Redis or Valkey, built for [a low barrier to entry](https://python-rq.org/). [Any Python function call](https://python-rq.org/docs/) can go on a queue, and there are [no queues, exchanges, or routing rules](https://python-rq.org/docs/#on-the-design) to set up first. Jobs are pickled, so RQ is Python-only. Put job functions in a module the worker [can import, not in `__main__`](https://python-rq.org/docs/#considerations-for-jobs), and run workers on the same source code as your app. Start workers with `rq worker --with-scheduler` to run jobs you schedule with `enqueue_in` or `enqueue_at`.
Huey is [a lightweight alternative](https://huey.readthedocs.io/en/latest/) with zero dependencies, and its queue can live in Redis, Postgres, SQLite, files, or memory. Its docs [match the storage to your workload](https://huey.readthedocs.io/en/latest/guide.html#storage-options): Redis for busy workloads, SQLite for moderate ones without a separate server, and Postgres when your app already runs it. Recurring tasks come built in with the `periodic_task()` decorator. For Django, Huey has [its own integration](https://huey.readthedocs.io/en/latest/contrib.html#django), and it provides [a production backend for Django's task framework](https://huey.readthedocs.io/en/latest/contrib.html#django-task-framework).
Dramatiq aims for [sane defaults for most SaaS workloads](https://dramatiq.io/motivation.html) on RabbitMQ or Redis. Mark a function with `@dramatiq.actor` and enqueue it with `.send()`, [passing only JSON-encodable arguments](https://dramatiq.io/guide.html). When an actor raises, Dramatiq [retries it with exponential backoff](https://dramatiq.io/guide.html#error-handling). It [acknowledges a message only after processing it](https://dramatiq.io/advanced.html#message-persistence), while Celery by default does so [just before running the task](https://docs.celeryq.dev/en/stable/userguide/tasks.html#Task.acks_late). For cron-style jobs, pair it with [a separate scheduler](https://dramatiq.io/cookbook.html#scheduling). Dramatiq is [licensed under the LGPL](https://dramatiq.io/).
Make tasks safe to run twice: after a worker failure, the same message [can arrive again](https://dramatiq.io/best_practices.html#retriable-actors). Pass a task the ID of a database row rather than the object, and [re-fetch it when the task runs](https://docs.celeryq.dev/en/stable/userguide/tasks.html#state), since old data leads to race conditions. In Django, enqueue a task only [after the transaction commits](https://docs.celeryq.dev/en/stable/userguide/tasks.html#database-transactions), with Celery's `delay_on_commit()` or Huey's `on_commit_task()`. With celery beat, Huey, or Taskiq, run [only one scheduler](https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html) for periodic tasks, or you'll get duplicate tasks. In production, run workers under [a process manager](https://python-rq.org/docs/workers/).