---
# === IDENTITY ===
id: software/patterns/batch-processing/2026
canonical_question: "What are the best batch processing patterns?"
aliases:
  - "How to implement batch processing"
  - "Batch job design patterns and best practices"
  - "Chunked data processing with error handling"
  - "Batch vs stream processing when to use"
entity_type: software_reference
domain: software > patterns > batch processing
region: global
jurisdiction: global
temporal_scope: 2020-2026

# === VERIFICATION ===
last_verified: 2026-02-24
confidence: 0.90
version: 1.0
first_published: 2026-02-24

# === TEMPORAL VALIDITY ===
temporal_validity:
  status: stable
  last_breaking_change: null
  next_review: 2026-08-23
  change_sensitivity: low

# === CONSTRAINTS ===
constraints:
  - "Every batch item must be idempotent -- re-processing the same item must produce the same result without side effects"
  - "Always persist progress checkpoints to durable storage so jobs can resume after failure"
  - "Never load the entire dataset into memory -- use cursor-based iteration or chunked reads"
  - "Dead letter queues are mandatory for production batches -- failed items must not block the rest"
  - "Chunk size must be tuned per workload: too small = overhead, too large = memory pressure and blast radius"

# === SKIP CONDITIONS ===
skip_this_unit_if:
  - condition: "You need sub-second latency on individual events"
    use_instead: "Stream processing (Kafka Streams, Flink, or event-driven architecture)"
  - condition: "You need to process a single request-response, not a collection"
    use_instead: "Standard synchronous API design patterns"

# === AGENT HINTS ===
inputs_needed:
  - key: dataset_size
    question: "How large is the dataset you need to process?"
    type: choice
    options: ["< 10K items", "10K-1M items", "1M-100M items", "> 100M items"]
  - key: failure_tolerance
    question: "What happens if a single item fails?"
    type: choice
    options: ["skip and continue", "retry then dead-letter", "fail entire batch"]
  - key: language
    question: "What language/runtime are you using?"
    type: choice
    options: ["Python", "Node.js", "Go", "Java/Spring Batch", "other"]

# === DISTRIBUTION ===
canonical_source: "https://knowledgelib.io/software/patterns/batch-processing/2026"
suggested_citation: "Source: knowledgelib.io -- AI Knowledge Library (verified 2026-02-24)"

# === RELATED UNITS ===
related_kos:
  related_to:
    - id: software/patterns/concurrency-parallelism/2026
      label: "Concurrency & Parallelism Patterns"
    - id: software/patterns/connection-pooling/2026
      label: "Connection Pooling"
    - id: software/system-design/message-queue-event-driven/2026
      label: "Message Queue & Event-Driven Architecture"
  solves:
    - id: software/patterns/error-handling-strategies/2026
      label: "Error Handling Strategies"
  often_confused_with:
    - id: software/patterns/database-migration-strategies/2026
      label: "Database Migration Strategies"

# === SOURCES (7 authoritative sources) ===
sources:
  - id: src1
    title: "Batch Processing Demystified: Tools, Challenges, and Solutions"
    author: Acceldata
    url: https://www.acceldata.io/blog/batch-processing-demystified-tools-challenges-and-solutions
    type: technical_blog
    published: 2025-03-15
    reliability: high
  - id: src2
    title: "Common Batch Patterns -- Spring Batch Reference"
    author: VMware / Spring
    url: https://docs.spring.io/spring-batch/reference/common-patterns.html
    type: official_docs
    published: 2025-11-01
    reliability: authoritative
  - id: src3
    title: "Understanding Idempotency: A Key to Reliable Data Pipelines"
    author: Airbyte
    url: https://airbyte.com/data-engineering-resources/idempotency-in-data-pipelines
    type: technical_blog
    published: 2025-06-10
    reliability: high
  - id: src4
    title: "Dead Letter Queues (DLQ): The Complete Developer-Friendly Guide"
    author: Software Engineer's Notes
    url: https://swenotes.com/2025/09/25/dead-letter-queues-dlq-the-complete-developer-friendly-guide/
    type: technical_blog
    published: 2025-09-25
    reliability: moderate_high
  - id: src5
    title: "Batch Processing vs Stream Processing: Key Differences"
    author: Airbyte
    url: https://airbyte.com/data-engineering-resources/batch-processing-vs-stream-processing
    type: technical_blog
    published: 2025-08-20
    reliability: high
  - id: src6
    title: "Building Reliable Reprocessing and Dead Letter Queues with Kafka"
    author: Uber Engineering
    url: https://www.uber.com/blog/reliable-reprocessing/
    type: technical_blog
    published: 2024-03-15
    reliability: high
  - id: src7
    title: "Design Patterns for Batch Processing in Financial Services"
    author: Databricks
    url: https://www.databricks.com/blog/design-patterns-batch-processing-financial-services
    type: technical_blog
    published: 2025-01-22
    reliability: high
---

# Batch Processing Patterns

## TL;DR

- **Bottom line**: Chunk work into resumable units, track progress in durable storage, handle failures idempotently, and route poison messages to a dead letter queue.
- **Key tool/command**: `cursor-based chunking + dead letter queue + idempotent upserts`
- **Watch out for**: Loading the entire dataset into memory -- this is the #1 cause of OOM kills in batch jobs.
- **Works with**: Any language/runtime. Patterns are language-agnostic; examples below cover Python, Node.js, and Go.

## Constraints
<!-- Agents: read this section before recommending any action from this unit.
     These are hard constraints that must not be violated. -->

- Every batch item must be idempotent -- re-processing the same item must produce the same result without side effects
- Always persist progress checkpoints to durable storage so jobs can resume after failure
- Never load the entire dataset into memory -- use cursor-based iteration or chunked reads
- Dead letter queues are mandatory for production batches -- failed items must not block the rest
- Chunk size must be tuned per workload: too small = overhead, too large = memory pressure and blast radius

## Quick Reference

| Pattern | Throughput | Resumability | Complexity | Best For |
|---|---|---|---|---|
| Sequential | Low | Easy (checkpoint per item) | Minimal | Small datasets (<10K), strict ordering |
| Parallel Chunks | High | Good (checkpoint per chunk) | Moderate | Medium datasets (10K-1M), independent items |
| MapReduce | Very High | Good (per-partition) | High | Huge datasets (>1M), aggregation/analytics |
| Micro-Batching | High | Moderate (per micro-batch) | Moderate | Near-real-time with batch semantics |
| Streaming (Kafka/Flink) | Very High | Built-in (offsets) | High | Continuous unbounded data, sub-second needs |
| Fan-Out/Fan-In | Very High | Good (per worker) | High | Heterogeneous tasks, cloud functions |
| Pipeline (multi-stage) | High | Per-stage checkpoints | Moderate | ETL with distinct transform steps |
| Retry + DLQ | N/A (overlay) | Excellent | Low (add-on) | Any pattern above -- handles poison messages |

## Decision Tree

```
START
|-- Dataset size < 10K items?
|   |-- YES -> Sequential processing with per-item checkpoint
|   +-- NO  v
|-- Items independent (no ordering dependency)?
|   |-- YES v
|   |   |-- Dataset < 1M items?
|   |   |   |-- YES -> Parallel Chunks (worker pool, chunk size 100-1000)
|   |   |   +-- NO  v
|   |   |-- Need aggregation/reduce step?
|   |   |   |-- YES -> MapReduce (partition, map, shuffle, reduce)
|   |   |   +-- NO  -> Fan-Out/Fan-In (distribute to workers, collect results)
|   +-- NO (ordering required) v
|-- Need near-real-time (<5s latency)?
|   |-- YES -> Micro-Batching (buffer 50-500ms, flush as batch)
|   +-- NO  -> Pipeline with stage-level checkpoints
+-- DEFAULT -> Parallel Chunks + Retry/DLQ overlay
```

## Step-by-Step Guide

### 1. Define chunk boundaries using cursor-based iteration

Never use OFFSET/LIMIT for large datasets -- it rescans rows. Use a cursor column (auto-increment ID or timestamp) to paginate. This ensures O(1) cost per page regardless of dataset size. [src1]

```sql
-- Cursor-based chunking: fetch next 1000 items after last processed ID
SELECT * FROM items
WHERE id > :last_processed_id
ORDER BY id ASC
LIMIT 1000;
```

**Verify**: Check that each chunk returns exactly LIMIT rows (except the final chunk), and no items are skipped or duplicated.

### 2. Implement idempotent processing with upserts

Every write operation must be safe to retry. Use UPSERT (INSERT ... ON CONFLICT) so reprocessing the same item overwrites rather than duplicates. [src3]

```sql
-- Idempotent upsert: safe to replay
INSERT INTO processed_items (id, result, processed_at)
VALUES (:id, :result, NOW())
ON CONFLICT (id) DO UPDATE SET
  result = EXCLUDED.result,
  processed_at = EXCLUDED.processed_at;
```

**Verify**: Run the same item twice -- the row count should not increase, and the result should be identical.

### 3. Add progress tracking with checkpoints

Store the last successfully processed cursor value in a durable checkpoint table. On restart, read the checkpoint and resume from that point. [src2]

```sql
-- Save checkpoint after each successful chunk
INSERT INTO batch_checkpoints (job_id, last_cursor, items_processed, updated_at)
VALUES (:job_id, :last_cursor, :count, NOW())
ON CONFLICT (job_id) DO UPDATE SET
  last_cursor = EXCLUDED.last_cursor,
  items_processed = batch_checkpoints.items_processed + EXCLUDED.items_processed,
  updated_at = EXCLUDED.updated_at;
```

**Verify**: Kill the job mid-run, restart it, and confirm it resumes from the checkpoint without reprocessing completed items.

### 4. Route failures to a dead letter queue

When an item fails after N retries, write it to a dead letter table/queue with the error details. The main batch continues processing remaining items. [src4]

```sql
-- Dead letter entry for failed items
INSERT INTO dead_letter_queue (job_id, item_id, payload, error_message, retry_count, failed_at)
VALUES (:job_id, :item_id, :payload, :error, :retries, NOW());
```

**Verify**: Intentionally corrupt one item in the batch -- confirm the batch completes and the bad item appears in the DLQ.

### 5. Add monitoring and alerting

Track batch metrics: items processed, items failed, elapsed time, throughput (items/sec). Alert when failure rate exceeds threshold (e.g., >5% of items fail). [src7]

```python
# Log batch summary
logger.info(f"Batch complete: {processed}/{total} items, "
            f"{failed} failed ({failed/total*100:.1f}%), "
            f"{elapsed:.1f}s, {processed/elapsed:.0f} items/sec")
```

**Verify**: Check logs after a batch run to confirm all five metrics are present.

## Code Examples

### Python: Chunked batch processor with retry and DLQ

> Full script: [python-chunked-batch-processor-with-retry-and-dlq.py](scripts/python-chunked-batch-processor-with-retry-and-dlq.py) (37 lines)

```python
# Input:  Database table with items to process
# Output: Processed items written back, failures in DLQ
import time
import psycopg2  # psycopg2==2.9.9
CHUNK_SIZE = 500
# ... (see full script)
```

### Node.js: Parallel chunk processor with concurrency control

> Full script: [node-js-parallel-chunk-processor-with-concurrency-.js](scripts/node-js-parallel-chunk-processor-with-concurrency-.js) (42 lines)

```javascript
// Input:  Array of item IDs or cursor-based DB query
// Output: Processed results, failures logged to DLQ
// p-limit@5.0.0
import pLimit from "p-limit";
const CHUNK_SIZE = 500;
# ... (see full script)
```

### Go: Fan-out/fan-in with worker pool and progress tracking

> Full script: [go-fan-out-fan-in-with-worker-pool-and-progress-tr.go](scripts/go-fan-out-fan-in-with-worker-pool-and-progress-tr.go) (65 lines)

```go
// Input:  Database rows fetched via cursor pagination
// Output: Processed results, errors sent to DLQ channel
package main
import (
    "context"
# ... (see full script)
```

## Anti-Patterns

### Wrong: Loading entire dataset into memory

```python
# BAD -- loads all rows into memory at once; OOM on large datasets
all_items = db.query("SELECT * FROM items").fetchall()
for item in all_items:
    process(item)
```

### Correct: Cursor-based chunked iteration

```python
# GOOD -- fetches one chunk at a time; constant memory usage
cursor = 0
while True:
    chunk = db.query("SELECT * FROM items WHERE id > %s ORDER BY id LIMIT 500", cursor)
    if not chunk:
        break
    for item in chunk:
        process(item)
    cursor = chunk[-1]["id"]
```

### Wrong: No progress tracking (restart = reprocess everything)

```python
# BAD -- no checkpoint; crash at item 999,999 means starting over
for item in get_all_items():
    process(item)
```

### Correct: Checkpoint after each chunk

```python
# GOOD -- resume from last successful chunk on restart
cursor = load_checkpoint(job_id) or 0
while True:
    chunk = fetch_chunk(cursor, 500)
    if not chunk:
        break
    for item in chunk:
        process(item)
    cursor = chunk[-1]["id"]
    save_checkpoint(job_id, cursor)
```

### Wrong: Non-idempotent processing (duplicates on retry)

```python
# BAD -- INSERT creates duplicates if item is retried
db.execute("INSERT INTO results (item_id, value) VALUES (%s, %s)", (item_id, value))
```

### Correct: Idempotent upsert

```python
# GOOD -- ON CONFLICT prevents duplicates on retry
db.execute("""
    INSERT INTO results (item_id, value, updated_at)
    VALUES (%s, %s, NOW())
    ON CONFLICT (item_id) DO UPDATE SET value = EXCLUDED.value, updated_at = NOW()
""", (item_id, value))
```

### Wrong: Failed items block the entire batch

```python
# BAD -- one poisoned item stops the whole batch
for item in chunk:
    result = process(item)  # unhandled exception halts everything
    save(result)
```

### Correct: Catch, retry, then dead-letter

```python
# GOOD -- failures go to DLQ; batch continues
for item in chunk:
    try:
        result = process(item)
        save(result)
    except Exception as e:
        if retries_exhausted(item):
            write_to_dlq(item, e)
        else:
            schedule_retry(item)
```

## Common Pitfalls

- **OFFSET/LIMIT pagination on large tables**: Performance degrades linearly -- OFFSET 1000000 scans 1M rows. Fix: use cursor-based pagination with `WHERE id > :last_id ORDER BY id LIMIT :size`. [src1]
- **Unbounded retry loops**: Retrying a poisoned message forever wastes resources and blocks progress. Fix: set `MAX_RETRIES` (typically 3-5) and route to DLQ after exhaustion. [src4]
- **Single-threaded bottleneck on CPU-bound work**: Processing items sequentially when they are independent wastes available cores. Fix: use worker pool with `CONCURRENCY = num_CPUs` for CPU-bound or `CONCURRENCY = 10-50` for I/O-bound work. [src1]
- **Chunk size too large for available memory**: A chunk of 100K rows with 10KB each = 1GB in memory per chunk. Fix: profile memory, start with 100-500 items per chunk, and tune up. [src2]
- **No backpressure between producer and consumer**: Producer fills an unbounded channel/queue faster than consumers can drain, causing OOM. Fix: use bounded channels/queues (`make(chan Item, bufferSize)` in Go, `asyncio.Queue(maxsize=N)` in Python). [src6]
- **Ignoring transaction boundaries**: Processing and checkpointing in separate transactions means a crash between them causes reprocessing. Fix: wrap chunk processing + checkpoint update in a single transaction. [src2]
- **Clock-based cursors with duplicate timestamps**: Using `WHERE created_at > :last_ts` skips items created in the same millisecond. Fix: use a monotonic column (auto-increment ID) as cursor, or compound cursor `(timestamp, id)`. [src3]

## When to Use / When Not to Use

| Use When | Don't Use When | Use Instead |
|---|---|---|
| Processing a bounded dataset (files, DB tables, API exports) | Data arrives continuously and needs <1s processing | Stream processing (Kafka Streams, Flink) |
| Items can tolerate minutes/hours of latency | User is waiting for a synchronous response | Request/response API pattern |
| You need atomicity per chunk (all-or-nothing within a chunk) | Each event triggers an independent side effect | Event-driven architecture |
| Cost efficiency matters -- batch infra is idle between runs | 24/7 low-latency processing is required | Always-on streaming consumers |
| Complex multi-step transformations (ETL/ELT) | Simple event routing or pub/sub fan-out | Message broker + consumer groups |
| Regulatory/compliance requires auditable batch runs with clear boundaries | Real-time fraud detection or alerting | Stream processing with CEP |

## Important Caveats

- Batch and stream are not mutually exclusive. Many production systems use Lambda architecture (batch layer for completeness + stream layer for speed). Choose based on latency requirements, not ideology. [src5]
- Cloud-native batch (AWS Batch, GCP Cloud Run Jobs, Azure Container Apps Jobs) auto-scales workers but charges per vCPU-second. Profile your chunk processing time to avoid cost surprises on large datasets.
- Spring Batch is the most mature framework for JVM batch processing with built-in chunk-oriented processing, skip/retry policies, and job restartability. If you are on the JVM, use it rather than building from scratch. [src2]
- Idempotency requires a natural or synthetic unique key per item. If your data lacks one, generate a deterministic hash from the item's content before processing. [src3]
- DLQ reprocessing should be a deliberate manual operation, not automatic -- automatic DLQ replay without fixing the root cause creates infinite loops. [src6]

## Related Units

- [Concurrency & Parallelism Patterns](/software/patterns/concurrency-parallelism/2026)
- [Connection Pooling](/software/patterns/connection-pooling/2026)
- [Error Handling Strategies](/software/patterns/error-handling-strategies/2026)
- [Database Migration Strategies](/software/patterns/database-migration-strategies/2026)
- [Message Queue & Event-Driven Architecture](/software/system-design/message-queue-event-driven/2026)