--- # === IDENTITY === id: software/system-design/job-task-queue/2026 canonical_question: "How do I design a distributed job and task queue system?" aliases: - "distributed task queue architecture design" - "how to build a job queue system at scale" - "Celery vs BullMQ vs Temporal vs Sidekiq comparison" - "system design interview: design a task queue" - "at-least-once delivery task processing architecture" - "background job processing system design" - "async task queue with retries and dead letter queue" - "designing reliable job scheduling infrastructure" entity_type: software_reference domain: software > system-design > job_task_queue region: global jurisdiction: global temporal_scope: 2020-2026 # === VERIFICATION === last_verified: 2026-02-23 confidence: 0.93 version: 1.0 first_published: 2026-02-23 # === TEMPORAL VALIDITY === temporal_validity: status: stable last_breaking_change: "Celery 5.6 Recovery release (2025); BullMQ 5.x with flows (2024); Temporal 1.x GA (2023)" next_review: 2026-08-22 change_sensitivity: medium # === CONSTRAINTS === constraints: - "All task handlers MUST be idempotent — at-least-once delivery means tasks can be executed more than once after retries or rebalancing" - "Never store large payloads (>64 KB) directly in the queue message — pass a reference (S3 key, DB ID) and fetch inside the worker" - "Always configure a dead letter queue (DLQ) — without one, poison messages block the entire queue or are silently dropped" - "Set explicit task TTLs and max retry counts — unbounded retries cause infinite loops and resource exhaustion" - "Broker persistence must match your durability requirements — Redis AOF/RDB for BullMQ, RabbitMQ with durable queues for Celery, Cassandra/PostgreSQL for Temporal" # === SKIP CONDITIONS === skip_this_unit_if: - condition: "User needs a message bus or event streaming system (Kafka, Pulsar, NATS) rather than a job queue" use_instead: "Event streaming / pub-sub architecture documentation" - condition: "User needs a cron scheduler only, not distributed task processing" use_instead: "OS-level cron, systemd timers, or cloud scheduler (AWS EventBridge, Cloud Scheduler)" - condition: "User needs a workflow orchestrator for multi-step data pipelines (Airflow, Prefect)" use_instead: "Data pipeline orchestration documentation" # === AGENT HINTS === inputs_needed: - key: language question: "What is your primary programming language?" type: choice options: ["Python", "Node.js / TypeScript", "Ruby", "Go", "Java"] - key: scale question: "What is your expected task throughput?" type: choice options: ["<1K tasks/day (small)", "1K-100K tasks/day (medium)", "100K-10M tasks/day (high)", ">10M tasks/day (extreme)"] - key: task_duration question: "How long do your typical tasks run?" type: choice options: ["<1 second (fast)", "1-60 seconds (medium)", "1-60 minutes (long)", ">1 hour (very long / workflows)"] - key: delivery_guarantee question: "What delivery guarantee do you need?" type: choice options: ["At-least-once (default, simplest)", "Exactly-once (requires idempotency layer)", "At-most-once (fire and forget)"] # === DISTRIBUTION === canonical_source: "https://knowledgelib.io/software/system-design/job-task-queue/2026" suggested_citation: "Source: knowledgelib.io — AI Knowledge Library (verified 2026-02-23)" # === RELATED UNITS === related_kos: related_to: - id: "software/debugging/redis-memory-issues/2026" label: "Redis Memory Issues" - id: "software/system-design/chat-app-at-scale/2026" label: "Chat App at Scale (uses queues for message fan-out)" solves: - id: "software/debugging/nodejs-memory-leaks/2026" label: "Node.js Memory Leaks (often caused by unbounded queue buffers)" - id: "software/debugging/kubernetes-pod-pending/2026" label: "Kubernetes Pod Pending (worker scaling issues)" often_confused_with: - id: "software/system-design/social-media-feed/2026" label: "Social Media Feed (fan-out pattern, not a job queue)" # === SOURCES (7 authoritative sources) === # Types: official_docs, technical_blog, rfc_spec, academic_paper, community_resource, industry_report # Reliability: high, moderate_high, moderate, moderate_low, low, authoritative sources: - id: src1 title: "Celery - Distributed Task Queue Documentation" author: Celery Project url: https://docs.celeryq.dev/en/stable/ type: official_docs published: 2025-12-01 reliability: authoritative - id: src2 title: "BullMQ Documentation — What is BullMQ" author: Taskforce.sh url: https://docs.bullmq.io/ type: official_docs published: 2025-10-15 reliability: authoritative - id: src3 title: "Temporal.io — Taming the Chaos of Distributed Systems" author: Temporal Technologies url: https://docs.temporal.io/ type: official_docs published: 2025-11-01 reliability: authoritative - id: src4 title: "Sidekiq Best Practices" author: Mike Perham url: https://github.com/sidekiq/sidekiq/wiki/Best-Practices type: community_resource published: 2025-06-01 reliability: high - id: src5 title: "Design a Distributed Job Scheduler: System Design Guide" author: System Design Handbook url: https://www.systemdesignhandbook.com/guides/design-a-distributed-job-scheduler/ type: technical_blog published: 2025-08-20 reliability: moderate_high - id: src6 title: "FOQS: Scaling a Distributed Priority Queue" author: Meta Engineering url: https://engineering.fb.com/2021/02/22/production-engineering/foqs-scaling-a-distributed-priority-queue/ type: technical_blog published: 2021-02-22 reliability: high - id: src7 title: "Dead Letter Queues (DLQ): The Complete Developer-Friendly Guide" author: Software Engineer's Notes url: https://swenotes.com/2025/09/25/dead-letter-queues-dlq-the-complete-developer-friendly-guide/ type: technical_blog published: 2025-09-25 reliability: moderate_high --- # Distributed Job & Task Queue System Design ## TL;DR - **Bottom line**: A distributed job queue decouples producers from workers via a broker, enabling asynchronous task processing with retries, dead letter queues, and horizontal scaling — choose Celery for Python, BullMQ for Node.js, Sidekiq for Ruby, or Temporal for long-running workflows. - **Key tool/command**: `celery -A tasks worker --loglevel=info` (Python) / `new Worker(queueName, processor)` (BullMQ/Node.js) - **Watch out for**: Non-idempotent task handlers — at-least-once delivery means your task will run more than once after retries, causing duplicate side effects. - **Works with**: Any language. Core patterns are language-agnostic. Code examples in Python (Celery) and Node.js/TypeScript (BullMQ). ## Constraints - All task handlers MUST be idempotent — at-least-once delivery means duplicate execution is expected, not exceptional [src1] - Never store large payloads (>64 KB) in queue messages — pass a reference (S3 key, DB row ID) and fetch in the worker [src4] - Always configure a dead letter queue (DLQ) — without one, poison messages block processing or are silently lost [src7] - Set explicit task TTLs and max retry counts — unbounded retries cause infinite loops and resource exhaustion [src5] - Broker persistence must match durability needs — an in-memory Redis with no AOF loses all pending jobs on crash [src2] - Never let workers auto-acknowledge before processing completes — premature ACK means lost tasks on worker crash [src1] ## Quick Reference | Component | Role | Technology Options | Scaling Strategy | |---|---|---|---| | Producer | Enqueues tasks with payload and metadata | Any application code (web server, CLI, cron) | Stateless; scale with application tier | | Message Broker | Stores and routes tasks to workers | Redis (BullMQ, Sidekiq), RabbitMQ (Celery), PostgreSQL (Temporal), SQS (cloud) | Cluster mode with replication; partition by queue | | Worker Pool | Pulls and executes tasks | Celery workers, BullMQ Workers, Sidekiq threads, Temporal workers | Horizontal scaling; auto-scale on queue depth | | Result Backend | Stores task outcomes and return values | Redis, PostgreSQL, MongoDB, S3 | Optional; omit if tasks are fire-and-forget | | Dead Letter Queue | Isolates poison messages after max retries | Dedicated queue in same broker | Monitor DLQ depth; alert on growth | | Scheduler | Triggers periodic/delayed tasks | Celery Beat, BullMQ repeatable jobs, cron, CloudWatch Events | Single-leader pattern to prevent duplicate scheduling | | Rate Limiter | Throttles task processing rate | Token bucket per queue or per worker group | Protects downstream APIs from overload | | Priority Router | Routes high-priority tasks before low-priority | Weighted queues (Sidekiq), priority levels (BullMQ) | Separate worker pools per priority tier | | Observability | Monitors queue depth, latency, failure rate | Prometheus + Grafana, Datadog, Flower (Celery) | Alert on queue depth > threshold, DLQ growth | | Retry Engine | Retries failed tasks with backoff | Built-in (all frameworks), custom exponential backoff | Cap retries (3-5); exponential backoff with jitter | ## Decision Tree > Full script: [decision-tree.txt](scripts/decision-tree.txt) (26 lines) ``` START: Choose a Task Queue Framework ├── Language: Python? │ ├── YES → Tasks run < 1 hour? │ │ ├── YES → Use Celery + Redis/RabbitMQ broker │ │ └── NO → Use Temporal Python SDK (durable execution) # ... (see full script) ``` ``` START: Choose Delivery Guarantee ├── Can your system tolerate duplicate processing? │ ├── YES → At-least-once (default for all frameworks — simplest) │ └── NO ↓ ├── Can you make handlers idempotent (dedup key, upsert, conditional write)? │ ├── YES → At-least-once + idempotency layer (recommended) │ └── NO ↓ ├── Is losing occasional tasks acceptable? │ ├── YES → At-most-once (ACK before processing — fast but lossy) │ └── NO ↓ └── DEFAULT → At-least-once + idempotent handlers (industry standard) ``` ## Step-by-Step Guide ### 1. Define the task interface and serialization Every task needs a name, a serializable payload, and metadata (retry count, priority, timeout). Keep payloads small and pass references to large data. [src1] ```python # Python — Celery task definition from celery import Celery app = Celery('tasks', broker='redis://localhost:6379/0', backend='redis://localhost:6379/1') @app.task(bind=True, max_retries=3, default_retry_delay=60, acks_late=True) def process_order(self, order_id: int): """Process a single order. Idempotent: checks order status before acting.""" order = db.get_order(order_id) # fetch by reference, not by value if order.status == 'processed': return # idempotent guard try: charge_payment(order) update_inventory(order) order.status = 'processed' db.save(order) except PaymentError as exc: raise self.retry(exc=exc) ``` **Verify**: `celery -A tasks inspect active` --> expected: task appears in active list when running ### 2. Configure the broker with persistence and DLQ Set up the message broker with appropriate durability. For Redis, enable AOF persistence. Configure dead letter routing for failed messages. [src7] ```python # Python — Celery configuration with DLQ and retry policy app.conf.update( task_acks_late=True, # ACK after processing, not before worker_prefetch_multiplier=1, # fetch one task at a time (fairness) task_reject_on_worker_lost=True, # re-queue if worker crashes mid-task task_default_queue='default', task_queues={ 'default': {'exchange': 'default', 'routing_key': 'default'}, 'priority': {'exchange': 'priority', 'routing_key': 'priority'}, 'dead_letter': {'exchange': 'dead_letter', 'routing_key': 'dead_letter'}, }, task_annotations={ 'tasks.process_order': {'rate_limit': '100/m'}, # max 100 per minute }, broker_transport_options={ 'visibility_timeout': 3600, # 1 hour before re-delivery }, ) ``` **Verify**: `redis-cli CONFIG GET appendonly` --> expected: `yes` (AOF enabled) ### 3. Implement worker pool with concurrency control Start workers with appropriate concurrency. Use prefork for CPU-bound tasks (Celery), event loop for I/O-bound tasks (BullMQ). [src1] ```bash # Start Celery workers with concurrency and queue binding celery -A tasks worker --loglevel=info --concurrency=4 -Q default,priority # Start a dedicated DLQ processor (manual inspection) celery -A tasks worker --loglevel=warning --concurrency=1 -Q dead_letter ``` **Verify**: `celery -A tasks inspect ping` --> expected: `pong` from all workers ### 4. Add retry logic with exponential backoff Configure exponential backoff with jitter to prevent thundering herd on transient failures. Cap maximum retries. [src5] ```python # Python — exponential backoff with jitter import random @app.task(bind=True, max_retries=5, acks_late=True) def call_external_api(self, endpoint: str, payload: dict): try: response = requests.post(endpoint, json=payload, timeout=30) response.raise_for_status() return response.json() except requests.RequestException as exc: # Exponential backoff: 2^retry * base + random jitter backoff = (2 ** self.request.retries) * 30 + random.randint(0, 30) raise self.retry(exc=exc, countdown=backoff) ``` **Verify**: Check retry count in Flower dashboard or `celery -A tasks inspect reserved` ### 5. Set up monitoring and alerting Monitor queue depth, processing latency, failure rate, and DLQ growth. Alert before queues back up. [src6] ```bash # Check queue depth (Redis / BullMQ) redis-cli LLEN bull:myqueue:wait # Check Celery queue depth celery -A tasks inspect active_queues # Flower web dashboard for Celery celery -A tasks flower --port=5555 ``` **Verify**: Navigate to `http://localhost:5555` --> expected: Flower dashboard with worker and task stats ### 6. Implement graceful shutdown and scaling Workers must finish in-progress tasks before shutting down. Use SIGTERM handling and drain mode. [src4] ```bash # Graceful shutdown: send SIGTERM, workers finish current tasks celery -A tasks control shutdown # Scale workers dynamically based on queue depth # Kubernetes HPA example: scale on custom metric (queue depth) # kubectl autoscale deployment celery-worker --min=2 --max=20 --cpu-percent=70 ``` **Verify**: Send SIGTERM to worker process --> expected: worker finishes current task, then exits cleanly ## Code Examples ### Python (Celery): Basic Task Producer and Consumer ```python # Input: order_id (int) — reference to order in database # Output: task result stored in Redis backend from celery import Celery app = Celery('shop', broker='redis://localhost:6379/0', backend='redis://localhost:6379/1') @app.task(bind=True, max_retries=3, acks_late=True) def process_order(self, order_id: int): """Idempotent order processor with retry.""" order = db.get(order_id) if order.status == 'done': return # idempotent guard try: charge(order) order.status = 'done' db.save(order) except Exception as exc: raise self.retry(exc=exc, countdown=60 * (2 ** self.request.retries)) # Producer side: enqueue the task result = process_order.delay(order_id=42) print(result.get(timeout=120)) # blocks until result ready ``` ### Node.js (BullMQ): Queue with Worker and Events > Full script: [node-js-bullmq-queue-with-worker-and-events.ts](scripts/node-js-bullmq-queue-with-worker-and-events.ts) (28 lines) ```typescript // Input: job data object { orderId: number } // Output: completed job with return value in Redis import { Queue, Worker, QueueEvents } from 'bullmq'; // bullmq@5.x import IORedis from 'ioredis'; // ioredis@5.x const connection = new IORedis({ host: '127.0.0.1', port: 6379, maxRetriesPerRequest: null }); # ... (see full script) ``` ## Anti-Patterns ### Wrong: Storing entire payload in the queue message ```python # BAD — large payload serialized into Redis, causes memory pressure and slow serialization @app.task def process_image(self, image_bytes: bytes, metadata: dict): # image_bytes could be 10 MB+ — broker was not designed for this result = resize(image_bytes) save(result) ``` ### Correct: Pass a reference, fetch data in the worker ```python # GOOD — only a lightweight reference travels through the broker @app.task def process_image(self, image_s3_key: str): image_bytes = s3.download(image_s3_key) # fetch in worker result = resize(image_bytes) save(result) ``` ### Wrong: Acknowledging tasks before processing completes ```python # BAD — default Celery acks early; if worker crashes mid-task, the task is lost app.conf.task_acks_late = False # default — ACK on receive, not on completion ``` ### Correct: ACK after successful processing ```python # GOOD — task is re-queued if worker dies before completing app.conf.task_acks_late = True app.conf.task_reject_on_worker_lost = True ``` ### Wrong: No retry limit — infinite retry loops ```python # BAD — poison message retries forever, consuming resources indefinitely @app.task(bind=True) def fragile_task(self, data): try: do_work(data) except Exception as exc: raise self.retry(exc=exc) # no max_retries — retries forever ``` ### Correct: Bounded retries with exponential backoff and DLQ routing ```python # GOOD — max 5 retries with exponential backoff, then moves to DLQ @app.task(bind=True, max_retries=5, acks_late=True) def safe_task(self, data): try: do_work(data) except Exception as exc: if self.request.retries >= self.max_retries: dead_letter_queue.send(data, error=str(exc)) return # stop retrying, route to DLQ raise self.retry(exc=exc, countdown=30 * (2 ** self.request.retries)) ``` ### Wrong: Non-idempotent task handlers ```python # BAD — if task retries, user gets charged twice @app.task def charge_user(user_id, amount): payment_service.charge(user_id, amount) # no dedup — runs again on retry ``` ### Correct: Idempotent handler with deduplication key ```python # GOOD — idempotency key prevents duplicate charges @app.task def charge_user(user_id, amount, idempotency_key): if payment_service.has_processed(idempotency_key): return # already processed — skip payment_service.charge(user_id, amount, idempotency_key=idempotency_key) ``` ### Wrong: Single queue for all task types ```python # BAD — slow 10-minute report generation blocks fast 100ms notification sends app.conf.task_default_queue = 'default' # Everything goes to 'default': notifications, reports, emails, imports ``` ### Correct: Separate queues by priority and task duration ```python # GOOD — fast tasks are never blocked by slow ones app.conf.task_routes = { 'tasks.send_notification': {'queue': 'fast'}, # <1s tasks 'tasks.generate_report': {'queue': 'slow'}, # >1min tasks 'tasks.process_payment': {'queue': 'critical'}, # must not be delayed } # Run separate worker pools per queue # celery -A tasks worker -Q fast --concurrency=10 # celery -A tasks worker -Q slow --concurrency=2 # celery -A tasks worker -Q critical --concurrency=4 ``` ## Common Pitfalls - **Poison messages blocking the queue**: A single malformed task that always fails can block consumers in an infinite retry loop, preventing all other tasks from processing. Fix: set `max_retries` and route to a dead letter queue after exhaustion. [src7] - **Thundering herd on retry**: When many tasks fail simultaneously (e.g., downstream outage), they all retry at the same time, overwhelming the recovering service. Fix: add random jitter to exponential backoff: `delay = base * 2^retry + random(0, base)`. [src5] - **Worker memory leaks from long-running processes**: Celery workers that process millions of tasks accumulate memory. Fix: set `worker_max_tasks_per_child=1000` to recycle worker processes after N tasks. [src1] - **Visibility timeout too short**: If a task takes longer than the broker's visibility timeout, the broker re-delivers it to another worker, causing duplicate execution. Fix: set `visibility_timeout` higher than your longest expected task duration. [src2] - **Premature result backend queries**: Calling `result.get()` immediately after `task.delay()` blocks the caller, defeating the purpose of async processing. Fix: use callbacks, webhooks, or polling with `result.ready()`. [src1] - **No monitoring on queue depth**: Queues silently back up until workers are overwhelmed and tasks time out. Fix: monitor queue depth (e.g., `LLEN bull:queue:wait` for BullMQ) and alert at thresholds. [src6] - **Single scheduler instance without leader election**: Running multiple Celery Beat instances causes duplicate periodic task scheduling. Fix: use `django-celery-beat` with database lock or a single Beat instance with health checks. [src1] - **Ignoring task serialization format**: Using Python pickle for Celery task serialization is a remote code execution vulnerability. Fix: use JSON serialization (`task_serializer='json'`) and never accept pickle from untrusted sources. [src1] ## Diagnostic Commands ```bash # Check Celery worker status celery -A tasks inspect ping # List active tasks across all workers celery -A tasks inspect active # Check queue depth (Redis-backed broker) redis-cli LLEN celery # Check BullMQ queue depth redis-cli LLEN bull:myqueue:wait # Monitor Celery in real-time (Flower) celery -A tasks flower --port=5555 # Check RabbitMQ queue depth rabbitmqctl list_queues name messages_ready messages_unacknowledged # Check Sidekiq queue stats via Redis redis-cli SCARD queues # View failed jobs in BullMQ redis-cli LRANGE bull:myqueue:failed 0 10 # Check broker connection from worker celery -A tasks inspect report ``` ## Version History & Compatibility | Framework | Current Version | Status | Key Changes | Notes | |---|---|---|---|---| | Celery 5.6 (Recovery) | 5.6.x | Current stable | Memory leak fixes (Python 3.11+), credential logging fix | Python 3.9-3.13; Redis or RabbitMQ broker | | BullMQ 5.x | 5.x | Current stable | Job flows with parent-child dependencies, rate limiting, deduplication | Node.js 18+; requires Redis 6.2+ | | Temporal 1.x | 1.24+ | GA | Durable execution, versioned workflows, schedules API | Go, Java, TypeScript, Python SDKs | | Sidekiq 7.x | 7.x | Current stable | Capsules (multiple thread configs), embedding, metrics | Ruby 2.7+; Redis 6.2+; Pro/Enterprise for rate limiting | | Amazon SQS | — | Managed service | Standard (at-least-once) and FIFO (exactly-once) queues | Max message size 256 KB; 14-day retention | | Google Cloud Tasks | — | Managed service | HTTP target tasks, rate limiting, scheduling | Max 1 MB payload; auto-retry with backoff | ## When to Use / When Not to Use | Use When | Don't Use When | Use Instead | |---|---|---| | Background jobs that don't need immediate response (emails, reports, image processing) | Tasks that must complete within the HTTP request-response cycle (<100 ms) | In-process function call or thread pool | | Workload spikes that exceed server capacity — queue absorbs bursts | Low, steady throughput that a single server handles easily | Direct function invocation | | Tasks that call unreliable external APIs and need retry with backoff | Purely in-memory computation with no I/O | Multiprocessing / thread pool | | Long-running workflows spanning minutes to days (Temporal) | Simple cron-like scheduling with no retries needed | OS cron, systemd timers | | Fan-out: one event triggers many independent tasks in parallel | Strict sequential processing with strong ordering guarantees | Event streaming (Kafka, Pulsar) | | Decoupled microservices communicating asynchronously | Real-time bidirectional communication (chat, gaming) | WebSockets, gRPC streaming | ## Important Caveats - At-least-once is the default guarantee for all major frameworks — exactly-once requires idempotent handlers plus a deduplication store (Redis SET NX, database upsert) - Redis-based brokers (BullMQ, Sidekiq, Celery+Redis) lose pending tasks on crash unless Redis persistence (AOF/RDB) is enabled — RabbitMQ offers stronger durability out of the box - Celery's pickle serializer is a remote code execution vector — always use JSON serialization in production (`task_serializer = 'json'`) - Temporal is fundamentally different from Celery/BullMQ/Sidekiq: it manages workflow state server-side with durable execution, making it the right choice for long-running, multi-step processes but overkill for simple fire-and-forget tasks - BullMQ requires `maxRetriesPerRequest: null` in the IORedis connection config for workers — omitting this causes timeout errors under load - Task queue is not the same as event streaming — if you need publish-subscribe, event replay, or log compaction, use Kafka/Pulsar/NATS instead ## Related Units - [Redis Memory Issues](/software/debugging/redis-memory-issues/2026) - [Chat App at Scale](/software/system-design/chat-app-at-scale/2026) - [Node.js Memory Leaks](/software/debugging/nodejs-memory-leaks/2026) - [Kubernetes Pod Pending](/software/debugging/kubernetes-pod-pending/2026) - [Social Media Feed (fan-out pattern)](/software/system-design/social-media-feed/2026)