From 4e020707e78ed2a9b2c9e8b3d2a7ec1bc710ef12 Mon Sep 17 00:00:00 2001 From: Vinta Chen Date: Sun, 27 Sep 2026 08:03:28 +0800 Subject: [PATCH] docs: add Task Queues category intro Task Queues had no intro, so its meta description fell back to generic text and readers got no guidance on which library to pick. Co-Authored-By: Claude --- website/data/category_intros/task-queues.md | 22 +++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 website/data/category_intros/task-queues.md diff --git a/website/data/category_intros/task-queues.md b/website/data/category_intros/task-queues.md new file mode 100644 index 00000000..0fcb8f62 --- /dev/null +++ b/website/data/category_intros/task-queues.md @@ -0,0 +1,22 @@ +Two questions pick a Python task queue: is your app async, and what's your broker? Taskiq is for asyncio. Celery takes RabbitMQ or Redis, RQ Redis, Huey SQLite. + +How to choose: + +- An asyncio app, like FastAPI: Taskiq +- A batteries-included queue on RabbitMQ, Redis, or Amazon SQS, or tasks sent from non-Python code: Celery +- The simplest job queue, on Redis or Valkey: RQ +- No broker server to run, with the queue in SQLite or the Postgres you already have: Huey +- A production backend for Django's task framework: Huey +- Retries with backoff and acks after processing by default, on RabbitMQ or Redis: Dramatiq + +Taskiq's docs call it [an asyncio Celery implementation](https://taskiq-python.github.io/): you mark tasks with `@broker.task` and send them with `await task.kiq()`. Since sending is async only, its docs [point fully synchronous projects to Celery or Dramatiq](https://taskiq-python.github.io/guide/). For production, its docs [highly recommend](https://taskiq-python.github.io/guide/getting-started.html) taskiq-aio-pika or taskiq-nats as the broker and taskiq-redis as the result backend. In a FastAPI app, [taskiq-fastapi](https://taskiq-python.github.io/framework_integrations/taskiq-with-fastapi.html) connects the broker to your app. + +Celery is a task queue [with batteries included](https://docs.celeryq.dev/en/stable/getting-started/first-steps-with-celery.html): [celery beat](https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html) runs periodic tasks, and [Django support](https://docs.celeryq.dev/en/stable/django/first-steps-with-django.html) comes out of the box. It runs on RabbitMQ, Redis, or Amazon SQS. Its docs call RabbitMQ [an excellent choice for production](https://docs.celeryq.dev/en/stable/getting-started/first-steps-with-celery.html#choosing-a-broker) and warn that Redis is more likely to lose data on a sudden shutdown or power failure. Its protocol has clients in other languages, so non-Python code can send tasks too. Celery [doesn't support Windows](https://docs.celeryq.dev/en/stable/getting-started/introduction.html), though. + +RQ is a simple job queue on Redis or Valkey, built for [a low barrier to entry](https://python-rq.org/). [Any Python function call](https://python-rq.org/docs/) can go on a queue, and there are [no queues, exchanges, or routing rules](https://python-rq.org/docs/#on-the-design) to set up first. Jobs are pickled, so RQ is Python-only. Put job functions in a module the worker [can import, not in `__main__`](https://python-rq.org/docs/#considerations-for-jobs), and run workers on the same source code as your app. Start workers with `rq worker --with-scheduler` to run jobs you schedule with `enqueue_in` or `enqueue_at`. + +Huey is [a lightweight alternative](https://huey.readthedocs.io/en/latest/) with zero dependencies, and its queue can live in Redis, Postgres, SQLite, files, or memory. Its docs [match the storage to your workload](https://huey.readthedocs.io/en/latest/guide.html#storage-options): Redis for busy workloads, SQLite for moderate ones without a separate server, and Postgres when your app already runs it. Recurring tasks come built in with the `periodic_task()` decorator. For Django, Huey has [its own integration](https://huey.readthedocs.io/en/latest/contrib.html#django), and it provides [a production backend for Django's task framework](https://huey.readthedocs.io/en/latest/contrib.html#django-task-framework). + +Dramatiq aims for [sane defaults for most SaaS workloads](https://dramatiq.io/motivation.html) on RabbitMQ or Redis. Mark a function with `@dramatiq.actor` and enqueue it with `.send()`, [passing only JSON-encodable arguments](https://dramatiq.io/guide.html). When an actor raises, Dramatiq [retries it with exponential backoff](https://dramatiq.io/guide.html#error-handling). It [acknowledges a message only after processing it](https://dramatiq.io/advanced.html#message-persistence), while Celery by default does so [just before running the task](https://docs.celeryq.dev/en/stable/userguide/tasks.html#Task.acks_late). For cron-style jobs, pair it with [a separate scheduler](https://dramatiq.io/cookbook.html#scheduling). Dramatiq is [licensed under the LGPL](https://dramatiq.io/). + +Make tasks safe to run twice: after a worker failure, the same message [can arrive again](https://dramatiq.io/best_practices.html#retriable-actors). Pass a task the ID of a database row rather than the object, and [re-fetch it when the task runs](https://docs.celeryq.dev/en/stable/userguide/tasks.html#state), since old data leads to race conditions. In Django, enqueue a task only [after the transaction commits](https://docs.celeryq.dev/en/stable/userguide/tasks.html#database-transactions), with Celery's `delay_on_commit()` or Huey's `on_commit_task()`. With celery beat, Huey, or Taskiq, run [only one scheduler](https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html) for periodic tasks, or you'll get duplicate tasks. In production, run workers under [a process manager](https://python-rq.org/docs/workers/).