---
# === IDENTITY ===
id: business/erp-integration/poison-message-handling/2026
canonical_question: "How do you handle poison messages - triage and replay of failed ERP integration messages?"
aliases:
  - "Poison message handling and dead letter queue triage for ERP integrations"
  - "How to triage and replay failed ERP integration messages from DLQ"
  - "Poison pill detection and remediation in enterprise middleware"
  - "DLQ triage workflow and message replay patterns for ERP"
entity_type: erp_integration
domain: business > erp-integration > poison-message-handling
region: global
jurisdiction: global
temporal_scope: 2024-2026

# === SYSTEM PROFILE ===
# Architecture pattern card — covers poison message handling across all major brokers and iPaaS platforms.
systems:
  - name: "Cross-Platform (AWS SQS, Azure Service Bus, Apache Kafka, RabbitMQ)"
    vendor: "Multiple (AWS, Microsoft, Apache, VMware)"
    version: "Current GA versions as of 2026"
    edition: "All editions"
    deployment: cloud
    api_surface: "REST, AMQP, Kafka Protocol"
  - name: "iPaaS Middleware (MuleSoft Anypoint MQ, Boomi Atom Queue, Workato)"
    vendor: "Multiple (Salesforce/MuleSoft, Boomi, Workato)"
    version: "Current GA versions as of 2026"
    edition: "All editions"
    deployment: cloud
    api_surface: "REST, Proprietary"

# === VERIFICATION ===
last_verified: 2026-03-07
confidence: 0.87
version: 1.0
first_published: 2026-03-07

# === TEMPORAL VALIDITY ===
temporal_validity:
  status: evolving
  last_breaking_change: "2024-06 — AWS SQS DLQ Redrive API GA (StartMessageMoveTask, CancelMessageMoveTask, ListMessageMoveTasks)"
  next_review: 2026-09-03
  change_sensitivity: medium

# === CONSTRAINTS ===
constraints:
  - "Poison messages must be detected within 3-5 delivery attempts — fewer than 3 causes premature DLQ routing of transient failures, more than 5 wastes processing capacity"
  - "DLQ retention varies by platform: AWS SQS max 14 days (message timestamp does NOT reset on DLQ move), Azure Service Bus unlimited (Premium tier), Kafka depends on topic retention config"
  - "Replay without idempotency guarantees will create duplicate records in the target ERP — every replayed message must carry an idempotency key"
  - "Replay ordering matters for dependent messages — replaying child records before parent records creates referential integrity violations"
  - "DLQ messages have no automatic cleanup on most platforms — unprocessed messages accumulate indefinitely unless TTL or purge policies are configured"
  - "MuleSoft Anypoint MQ DLQ must be in the same region as the source queue — cross-region DLQ assignment is not supported"
  - "Azure Service Bus DLQ is a sub-queue of the parent entity (path: <queue>/$deadletterqueue) — it cannot be independently deleted or have its own DLQ"

# === SKIP CONDITIONS ===
skip_this_unit_if:
  - condition: "User needs general retry strategies and circuit breaker patterns (not poison-message-specific)"
    use_instead: "business/erp-integration/error-handling-dead-letter-queues/2026"
  - condition: "User needs idempotency key implementation for replay safety"
    use_instead: "business/erp-integration/idempotency-erp-integrations/2026"
  - condition: "User needs distributed transaction coordination across multiple ERPs"
    use_instead: "business/erp-integration/saga-pattern-erp-transactions/2026"

# === AGENT HINTS ===
inputs_needed:
  - key: message_broker
    question: "Which message broker or middleware are you using?"
    type: choice
    options:
      - "AWS SQS / SNS"
      - "Azure Service Bus"
      - "Apache Kafka"
      - "RabbitMQ"
      - "iPaaS (MuleSoft, Boomi, Workato)"
      - "Custom / other"
  - key: failure_type
    question: "What type of failures are you seeing?"
    type: choice
    options:
      - "Data quality errors (schema violations, missing required fields)"
      - "Transient errors (timeouts, rate limits, network issues)"
      - "Permanent errors (invalid API endpoint, authentication failures)"
      - "Mixed / unknown"
  - key: replay_requirement
    question: "Do you need to replay failed messages after fixing the root cause?"
    type: choice
    options:
      - "Yes — zero-loss, every message must eventually succeed"
      - "Yes — but only after manual review and approval"
      - "No — log and discard is acceptable"

# === DISTRIBUTION ===
canonical_source: "https://knowledgelib.io/business/erp-integration/poison-message-handling/2026"
suggested_citation: "Source: knowledgelib.io — AI Knowledge Library (verified 2026-03-07)"

# === RELATED UNITS ===
related_kos:
  depends_on:
    - id: "business/erp-integration/error-handling-dead-letter-queues/2026"
      label: "Error handling and DLQ fundamentals — retry strategies, exponential backoff, circuit breakers"
  related_to:
    - id: "business/erp-integration/idempotency-erp-integrations/2026"
      label: "Idempotency patterns — required for safe message replay"
    - id: "business/erp-integration/change-data-capture-erp/2026"
      label: "Change data capture — upstream event source that generates messages"
  solves:
    - id: "business/erp-integration/order-to-cash-integration/2026"
      label: "Order-to-cash integration — poison messages block order processing"
    - id: "business/erp-integration/procure-to-pay-integration/2026"
      label: "Procure-to-pay integration — invoice failures require triage and replay"
  alternative_to:
    - id: "business/erp-integration/saga-pattern-erp-transactions/2026"
      label: "Saga pattern — for coordinated multi-step compensation, not message-level triage"
  often_confused_with:
    - id: "business/erp-integration/error-handling-dead-letter-queues/2026"
      label: "Error handling & DLQ (covers retry/backoff/circuit breaker; this card covers what happens AFTER messages reach the DLQ)"

# === SOURCES ===
sources:
  - id: src1
    title: "Dead Letter Queue (DLQ): Architecture, Examples, and Best Practices (2026 Guide)"
    author: SRE School
    url: https://sreschool.com/blog/dead-letter-queue-dlq/
    type: technical_blog
    published: 2026-01-15
    reliability: high
  - id: src2
    title: "Dead Letter Channel — Enterprise Integration Patterns"
    author: Gregor Hohpe & Bobby Woolf
    url: https://www.enterpriseintegrationpatterns.com/patterns/messaging/DeadLetterChannel.html
    type: technical_blog
    published: 2003-10-01
    reliability: authoritative
  - id: src3
    title: "Service Bus Dead-Letter Queues — Azure Service Bus"
    author: Microsoft
    url: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dead-letter-queues
    type: official_docs
    published: 2025-05-15
    reliability: authoritative
  - id: src4
    title: "Using Dead-Letter Queues in Amazon SQS"
    author: Amazon Web Services
    url: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
    type: official_docs
    published: 2025-08-01
    reliability: authoritative
  - id: src5
    title: "Error Handling via Dead Letter Queue in Apache Kafka"
    author: Kai Waehner
    url: https://www.kai-waehner.de/blog/2022/05/30/error-handling-via-dead-letter-queue-in-apache-kafka/
    type: technical_blog
    published: 2022-05-30
    reliability: high
  - id: src6
    title: "Error Handling with Anypoint MQ — MuleSoft Blog"
    author: MuleSoft
    url: https://blogs.mulesoft.com/dev-guides/how-to-tutorials/error-handling-with-anypoint-mq/
    type: technical_blog
    published: 2024-03-15
    reliability: high
  - id: src7
    title: "Message Reprocessing: How to Implement the Dead Letter Queue"
    author: Redpanda
    url: https://www.redpanda.com/blog/reliable-message-processing-with-dead-letter-queue
    type: technical_blog
    published: 2025-04-10
    reliability: high
---

# Poison Message Handling: Triage and Replay of Failed ERP Integration Messages

## TL;DR

- **Bottom line**: A poison message is any message that repeatedly fails processing and blocks the queue. Detect within 3-5 delivery attempts, classify the error (transient vs permanent vs data quality), route to a typed DLQ, then triage via automated classification before manual review. Never replay without fixing the root cause and verifying idempotency. [src1, src2]
- **Key limit**: DLQ retention is finite on most platforms — AWS SQS max 14 days (original enqueue timestamp preserved), Azure Service Bus unlimited on Premium, Kafka depends on topic retention config. Unprocessed poison messages silently expire. [src3, src4]
- **Watch out for**: Infinite retry loops are the #1 anti-pattern — a message that fails due to a schema violation will fail identically on every retry, consuming processing capacity and blocking healthy messages behind it. Set maxDeliveryCount/maxReceiveCount to 3-5, not 100. [src1, src7]
- **Best for**: Any ERP integration where failed messages must be triaged, diagnosed, fixed, and replayed — order-to-cash, AP automation, inventory sync, payroll feeds, intercompany settlement.
- **Authentication**: N/A (pattern-level card). See system-specific cards for broker/iPaaS authentication details.

## System Profile

This card covers poison message handling as a cross-platform architecture pattern for ERP integrations. It focuses specifically on what happens *after* a message exhausts its retry budget and lands in a dead letter queue — detection, classification, triage, remediation, and replay. For the retry strategies that determine *when* a message becomes a poison message (exponential backoff, circuit breakers), see the companion card on error handling and DLQ fundamentals.

The patterns apply across all major message brokers (AWS SQS, Azure Service Bus, Apache Kafka, RabbitMQ) and iPaaS platforms (MuleSoft Anypoint MQ, Boomi Atom Queue, Workato, Celigo). The specific ERP system at either end of the integration (Salesforce, SAP, Oracle, NetSuite, Dynamics 365, Workday) does not change the poison message handling approach — it changes the error codes and data mapping fixes needed during remediation.

| System | Role | API Surface | Direction |
|---|---|---|---|
| Source ERP (e.g., Salesforce) | Event producer — generates change events or outbound messages | REST, Platform Events, CDC | Outbound |
| Message Broker (e.g., AWS SQS, Kafka) | Message transport + DLQ infrastructure | SQS API, Kafka Protocol | Transport |
| iPaaS (e.g., MuleSoft, Boomi) | Integration orchestrator — message transformation and routing | Anypoint MQ, Atom Queue | Orchestrator |
| Target ERP (e.g., SAP S/4HANA) | Message consumer — processes inbound records | OData, BAPI, IDoc | Inbound |

## API Surfaces & Capabilities

Poison message handling capabilities vary significantly across platforms. The key differentiators are automatic DLQ routing, DLQ inspection APIs, and native replay/redrive support: [src3, src4, src5]

| Platform | DLQ Type | Auto-Route | Max Delivery Count | Inspection API | Native Replay | DLQ Retention |
|---|---|---|---|---|---|---|
| AWS SQS | Separate queue | Yes (redrive policy) | Configurable (1-1000) | ReceiveMessage on DLQ | Yes (DLQ Redrive API) | Same as source (max 14 days) |
| Azure Service Bus | Sub-queue ($deadletterqueue) | Yes (MaxDeliveryCount) | Default 10, configurable | Peek/receive on sub-queue | Manual (receive + re-send) | Unlimited (Premium) |
| Apache Kafka | Separate topic (DLT) | Application-level | Application-level | Consumer on DLT topic | Application-level | Topic retention config |
| RabbitMQ | Separate queue (x-dead-letter-exchange) | Yes (x-delivery-limit) | Configurable via quorum queues | AMQP consume on DLQ | Manual (consume + re-publish) | Queue TTL config |
| MuleSoft Anypoint MQ | Separate queue | Yes (max delivery attempts) | Configurable | Anypoint MQ API | Yes (REM — Resubmit Error Messages) | 7 days default |
| Boomi Atom Queue | Built-in DLQ | Yes (after 7 attempts) | 7 (6 retries + original) | Queue Management panel | Yes (resend dead letters) | Atom storage lifecycle |

## Rate Limits & Quotas

### DLQ Throughput Limits

| Platform | Replay Rate Limit | Concurrent Replays | Max DLQ Size | Notes |
|---|---|---|---|---|
| AWS SQS | System-optimized or custom max velocity | 1 active redrive task per source queue | No hard limit (cost-based) | Redrive task max duration: 36 hours; max 100 active tasks per account [src4] |
| Azure Service Bus | No built-in rate limit on replay | N/A (manual process) | Entity size limit (Premium: 80 GB) | No automatic cleanup — messages persist until explicitly completed [src3] |
| Apache Kafka | Consumer throughput | Consumer group parallelism | Topic retention (size or time) | No native redrive — must implement consumer that reads DLT and produces to main topic [src5] |
| MuleSoft Anypoint MQ | API rate limits apply | Per-queue basis | 120,000 in-flight messages | REM feature provides managed replay with visibility [src6] |
| Boomi | Queue throughput | Per-atom basis | Atom storage capacity | Dead letters visible in Queue Management panel; batch resend available [src6] |

### Monitoring Thresholds

| Metric | Target | Alert When | Source |
|---|---|---|---|
| DLQ ingestion rate | < 1% of incoming throughput | Sustained > 1% for 15 minutes | [src1] |
| DLQ backlog (depth) | < 1,000 messages or team triage capacity | Growing for > 1 hour without corresponding triage | [src1] |
| Oldest message age in DLQ | < 24 hours for critical streams | Any message > 24 hours untriaged | [src1] |
| Replay success rate | > 95% | Below 90% on any replay batch | [src1] |
| Poison ratio (DLQ / total processed) | < 5% | Above 5% sustained | [src1] |
| Time to first triage | < 4 hours (critical), < 24 hours (standard) | Exceeding SLA threshold | [src1] |

## Authentication

N/A — this is a pattern-level card. Authentication is handled at the broker/iPaaS layer:

| Platform | Auth Method | Notes |
|---|---|---|
| AWS SQS | IAM roles / policies | DLQ access requires sqs:ReceiveMessage + sqs:DeleteMessage + sqs:SendMessage (for redrive) on both source and DLQ |
| Azure Service Bus | Shared Access Signature (SAS) or Azure AD (RBAC) | DLQ is a sub-queue — same connection string, append /$deadletterqueue to entity path [src3] |
| Apache Kafka | SASL/SCRAM, mTLS, or ACLs | DLT is a regular topic — requires separate ACL for consumer group reading DLT [src5] |
| MuleSoft | Anypoint Platform credentials + client ID/secret | DLQ management via Anypoint MQ API requires Manage Queues permission [src6] |

## Constraints
<!-- Agents: read this section before recommending any poison message handling approach from this unit.
     These are hard constraints that must not be violated. -->

- **Detection threshold**: Set maxDeliveryCount / maxReceiveCount to 3-5. Below 3 sends transient failures to DLQ prematurely; above 5 wastes processing capacity on truly unrecoverable messages. [src1, src7]
- **Retention is finite**: AWS SQS DLQ messages retain their *original* enqueue timestamp — a message with 14-day retention that spent 10 days in the source queue has only 4 days left in the DLQ before silent expiration. [src4]
- **Replay ordering**: Replaying messages out of order creates referential integrity violations — parent records (e.g., customer, sales order header) must be replayed before child records (e.g., invoice line items, shipment details).
- **Idempotency is mandatory for replay**: Every replayed message must carry an idempotency key. Without it, replay creates duplicate records in the target ERP. See the idempotency patterns card. [src1]
- **DLQ-of-DLQ is an anti-pattern**: If your DLQ consumer fails, do NOT route to a second DLQ. Log, alert, and stop processing — investigate the triage logic itself. [src1]
- **No cross-region DLQ on MuleSoft**: Anypoint MQ requires DLQ and source queue in the same region. [src6]
- **Azure DLQ has no TTL**: Messages in Azure Service Bus DLQ persist indefinitely (no time-to-live enforcement). Without a purge process, DLQ grows unbounded. [src3]

## Integration Pattern Decision Tree

```
START — Message has failed processing and landed in DLQ
├── Step 1: Classify the failure
│   ├── Transient error? (timeout, 429, 503, network error)
│   │   ├── YES → Should NOT be in DLQ — investigate why retries exhausted
│   │   │   ├── maxDeliveryCount too low? → Increase to 3-5
│   │   │   ├── Backoff delay too short? → Increase max backoff
│   │   │   └── Upstream system was down for extended period? → Expected; replay now
│   │   └── Action: REPLAY IMMEDIATELY (system has recovered)
│   ├── Data quality error? (schema violation, missing field, invalid reference)
│   │   ├── YES → Can the message be fixed automatically?
│   │   │   ├── YES → Auto-remediate (default values, field mapping fix) → REPLAY
│   │   │   └── NO → Route to manual review queue
│   │   └── Action: FIX DATA → REPLAY WITH IDEMPOTENCY CHECK
│   ├── Permanent error? (invalid endpoint, auth failure, business rule violation)
│   │   ├── YES → Is it a code/config bug?
│   │   │   ├── YES → Fix code/config, deploy → REPLAY ENTIRE BATCH
│   │   │   └── NO → Business rule rejection (e.g., duplicate PO, closed period)
│   │   │       ├── Fixable in target ERP? → Fix target state → REPLAY
│   │   │       └── Not fixable? → DISCARD + ALERT + LOG for audit
│   │   └── Action: FIX ROOT CAUSE → GATED REPLAY
│   └── Unknown error?
│       └── Action: QUARANTINE → MANUAL TRIAGE → determine category above
├── Step 2: Remediate
│   ├── Automated fix possible? → Apply transform → validate → stage for replay
│   └── Manual fix needed? → Alert ops team → ticket created → SLA clock starts
├── Step 3: Replay
│   ├── Verify idempotency key is present
│   ├── Verify ordering constraints (parent before child)
│   ├── Replay to original queue (NOT directly to consumer)
│   ├── Monitor replay success rate
│   └── If replay fails again → QUARANTINE (do NOT infinite-loop)
└── Step 4: Post-mortem
    ├── Was this a new failure category? → Add classifier rule
    ├── Recurring pattern? → Fix upstream validation
    └── Update monitoring thresholds if needed
```

## Quick Reference

| Scenario | Action | Replay? | Idempotency Required? | Alert Level |
|---|---|---|---|---|
| Schema violation (missing required field) | Fix data, validate, replay | Yes | Yes | Warning |
| Invalid foreign key reference | Create parent record first, then replay | Yes (ordered) | Yes | Warning |
| Rate limit exhaustion (429) | Should not be in DLQ — increase retry budget | Yes (immediate) | Yes | Info |
| Authentication failure (401/403) | Fix credentials/permissions, replay batch | Yes | Yes | Critical |
| Business rule violation (e.g., closed period) | Fix target ERP state or discard | Conditional | Yes | Warning |
| Malformed payload (unparseable JSON/XML) | Discard — cannot be fixed automatically | No | N/A | Error |
| Target system permanently decommissioned | Discard + archive for audit | No | N/A | Critical |
| Duplicate record conflict (409) | Already processed — safe to discard | No | N/A | Info |
| Cascading failure (parent failed, children queued) | Fix parent first, then replay children in order | Yes (ordered) | Yes | Warning |
| Unknown/unclassified error | Quarantine for manual investigation | Pending triage | Yes | Error |

## Step-by-Step Integration Guide

### 1. Classify errors at the consumer level

Before a message ever reaches the DLQ, classify the error type in your consumer. This metadata travels with the message to the DLQ and determines the triage path. [src1, src7]

```python
# Error classification function — call in your consumer's catch block
def classify_error(exception, message):
    """Classify processing errors to determine DLQ triage path."""
    error_info = {
        "error_class": type(exception).__name__,
        "error_message": str(exception)[:500],
        "timestamp": datetime.utcnow().isoformat(),
        "message_id": message.get("message_id"),
        "attempt_count": message.get("approximate_receive_count", 0),
    }

    if isinstance(exception, (TimeoutError, ConnectionError)):
        error_info["category"] = "transient"
        error_info["retry_eligible"] = True
    elif isinstance(exception, (ValidationError, SchemaError)):
        error_info["category"] = "data_quality"
        error_info["retry_eligible"] = False  # fix data first
    elif isinstance(exception, (AuthenticationError, PermissionError)):
        error_info["category"] = "permanent"
        error_info["retry_eligible"] = False
    else:
        error_info["category"] = "unknown"
        error_info["retry_eligible"] = False

    return error_info
```

**Verify**: Check DLQ messages have `category` attribute set -> confirms classification is running.

### 2. Configure platform-specific DLQ routing

Set up automatic dead-letter routing with appropriate delivery count thresholds. [src3, src4]

```bash
# AWS SQS — Create DLQ and attach redrive policy
aws sqs create-queue --queue-name erp-orders-dlq \
  --attributes '{"MessageRetentionPeriod":"1209600"}'

aws sqs set-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/erp-orders \
  --attributes '{
    "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:123456789:erp-orders-dlq\",\"maxReceiveCount\":\"5\"}"
  }'

# Azure Service Bus — Set MaxDeliveryCount (default 10, recommend 5 for ERP)
az servicebus queue update \
  --resource-group erp-integration \
  --namespace-name erp-bus \
  --name erp-orders \
  --max-delivery-count 5
```

**Verify**: Send a message that always fails -> confirm it appears in DLQ after 5 attempts.

### 3. Build the DLQ triage consumer

Create a dedicated consumer that reads from the DLQ, classifies messages, and routes them through the triage workflow. [src1, src7]

```python
# DLQ triage consumer — reads, classifies, and routes poison messages
import json
import boto3
from datetime import datetime, timedelta

sqs = boto3.client("sqs")
DLQ_URL = "https://sqs.us-east-1.amazonaws.com/123456789/erp-orders-dlq"
SOURCE_URL = "https://sqs.us-east-1.amazonaws.com/123456789/erp-orders"

def triage_dlq_messages(max_messages=10):
    """Read DLQ, classify, and route for remediation or replay."""
    response = sqs.receive_message(
        QueueUrl=DLQ_URL,
        MaxNumberOfMessages=max_messages,
        MessageAttributeNames=["All"],
        AttributeNames=["All"],
    )

    for msg in response.get("Messages", []):
        body = json.loads(msg["Body"])
        error_category = msg.get("MessageAttributes", {}).get(
            "error_category", {}
        ).get("StringValue", "unknown")
        receive_count = int(msg["Attributes"].get("ApproximateReceiveCount", 0))

        # Prevent infinite triage loops
        if receive_count > 3:
            quarantine_message(msg, reason="triage_loop_detected")
            continue

        if error_category == "transient":
            # Transient errors in DLQ = system recovered, safe to replay
            replay_message(msg, body, SOURCE_URL)
        elif error_category == "data_quality":
            attempt_auto_fix(msg, body)
        elif error_category == "permanent":
            route_to_manual_review(msg, body)
        else:
            quarantine_message(msg, reason="unclassified")
```

**Verify**: `aws sqs get-queue-attributes --queue-url $DLQ_URL --attribute-names ApproximateNumberOfMessages` -> count decreasing as triage runs.

### 4. Implement safe replay with idempotency check

Replay messages back to the source queue with idempotency verification to prevent duplicate processing. [src1, src4]

```python
def replay_message(dlq_msg, body, target_queue_url):
    """Replay a DLQ message with idempotency safety."""
    idempotency_key = body.get("idempotency_key")
    if not idempotency_key:
        quarantine_message(dlq_msg, reason="missing_idempotency_key")
        return

    # Check if this message was already successfully processed
    if is_already_processed(idempotency_key):
        # Safe to delete from DLQ — already handled
        sqs.delete_message(
            QueueUrl=DLQ_URL,
            ReceiptHandle=dlq_msg["ReceiptHandle"],
        )
        log_replay_event(idempotency_key, "skipped_already_processed")
        return

    # Add replay metadata
    body["_replay"] = {
        "replayed_at": datetime.utcnow().isoformat(),
        "original_dlq_message_id": dlq_msg["MessageId"],
        "replay_attempt": body.get("_replay", {}).get("replay_attempt", 0) + 1,
    }

    # Max replay attempts — prevent infinite replay loops
    if body["_replay"]["replay_attempt"] > 3:
        quarantine_message(dlq_msg, reason="max_replay_attempts_exceeded")
        return

    # Re-send to source queue
    sqs.send_message(
        QueueUrl=target_queue_url,
        MessageBody=json.dumps(body),
        MessageAttributes={
            "idempotency_key": {
                "DataType": "String",
                "StringValue": idempotency_key,
            },
            "is_replay": {
                "DataType": "String",
                "StringValue": "true",
            },
        },
    )

    # Delete from DLQ only after successful re-send
    sqs.delete_message(
        QueueUrl=DLQ_URL,
        ReceiptHandle=dlq_msg["ReceiptHandle"],
    )
    log_replay_event(idempotency_key, "replayed_successfully")
```

**Verify**: Replay a known-good message -> confirm it processes in source queue, verify no duplicate in target ERP.

## Code Examples

### Python: DLQ Depth Monitoring with CloudWatch Alerting

```python
# Input:  DLQ queue URL, SNS topic ARN for alerts
# Output: CloudWatch alarm that fires when DLQ depth exceeds threshold

import boto3

cloudwatch = boto3.client("cloudwatch")

def create_dlq_depth_alarm(queue_name, threshold=100, sns_topic_arn=None):
    """Create CloudWatch alarm for DLQ message depth."""
    cloudwatch.put_metric_alarm(
        AlarmName=f"dlq-depth-{queue_name}",
        AlarmDescription=f"DLQ {queue_name} has > {threshold} messages",
        Namespace="AWS/SQS",
        MetricName="ApproximateNumberOfMessagesVisible",
        Dimensions=[{"Name": "QueueName", "Value": queue_name}],
        Statistic="Maximum",
        Period=300,  # 5-minute check
        EvaluationPeriods=2,  # 2 consecutive breaches
        Threshold=threshold,
        ComparisonOperator="GreaterThanThreshold",
        AlarmActions=[sns_topic_arn] if sns_topic_arn else [],
        TreatMissingData="notBreaching",
    )

def create_dlq_age_alarm(queue_name, max_age_seconds=86400, sns_topic_arn=None):
    """Alert when oldest DLQ message exceeds age threshold (default 24h)."""
    cloudwatch.put_metric_alarm(
        AlarmName=f"dlq-age-{queue_name}",
        AlarmDescription=f"DLQ {queue_name} has messages older than {max_age_seconds}s",
        Namespace="AWS/SQS",
        MetricName="ApproximateAgeOfOldestMessage",
        Dimensions=[{"Name": "QueueName", "Value": queue_name}],
        Statistic="Maximum",
        Period=300,
        EvaluationPeriods=1,
        Threshold=max_age_seconds,
        ComparisonOperator="GreaterThanThreshold",
        AlarmActions=[sns_topic_arn] if sns_topic_arn else [],
    )
```

### JavaScript/Node.js: Kafka DLT Consumer with Triage Logic

```javascript
// Input:  Kafka connection config, DLT topic name
// Output: Triage consumer that classifies and routes failed messages

const { Kafka } = require("kafkajs"); // kafkajs@2.2.4

const kafka = new Kafka({ brokers: ["broker:9092"] });
const consumer = kafka.consumer({ groupId: "dlq-triage" });
const producer = kafka.producer({ idempotent: true });

async function runDLTTriageConsumer(dltTopic, mainTopic) {
  await consumer.connect();
  await producer.connect();
  await consumer.subscribe({ topic: dltTopic, fromBeginning: false });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      const headers = message.headers || {};
      const errorType = headers["error-type"]?.toString() || "unknown";
      const retryCount = parseInt(headers["retry-count"]?.toString() || "0");
      const idempotencyKey = headers["idempotency-key"]?.toString();

      if (!idempotencyKey) {
        console.error(`DLT message missing idempotency key: ${message.key}`);
        await logToQuarantine(message, "missing_idempotency_key");
        return; // commit offset, do not replay
      }

      if (retryCount > 3) {
        await logToQuarantine(message, "max_retries_exceeded");
        return;
      }

      switch (errorType) {
        case "transient":
          // System recovered — replay to main topic
          await producer.send({
            topic: mainTopic,
            messages: [{
              key: message.key,
              value: message.value,
              headers: {
                ...headers,
                "is-replay": "true",
                "replay-timestamp": new Date().toISOString(),
                "retry-count": String(retryCount + 1),
              },
            }],
          });
          break;
        case "data_quality":
          await routeToRemediationTopic(message);
          break;
        case "permanent":
          await logToQuarantine(message, errorType);
          await alertOpsTeam(message, errorType);
          break;
        default:
          await logToQuarantine(message, "unclassified");
      }
    },
  });
}
```

### cURL: Azure Service Bus DLQ Inspection

```bash
# Input:  Service Bus namespace, queue name, SAS token
# Output: Peek at dead-lettered messages for triage

# Generate SAS token (or use Azure CLI: az servicebus queue authorization-rule keys list)
SAS_TOKEN="SharedAccessSignature sr=..."

# Peek messages in DLQ (non-destructive)
curl -X POST \
  "https://erp-bus.servicebus.windows.net/erp-orders/\$deadletterqueue/messages/head?timeout=30" \
  -H "Authorization: $SAS_TOKEN" \
  -H "Content-Type: application/atom+xml;type=entry;charset=utf-8"

# Read and lock a DLQ message (destructive — locks for processing)
curl -X POST \
  "https://erp-bus.servicebus.windows.net/erp-orders/\$deadletterqueue/messages/head?timeout=30" \
  -H "Authorization: $SAS_TOKEN" \
  -D -  # capture response headers including lock token

# Complete (delete) a DLQ message after successful triage
curl -X DELETE \
  "https://erp-bus.servicebus.windows.net/erp-orders/\$deadletterqueue/messages/{messageId}/{lockToken}" \
  -H "Authorization: $SAS_TOKEN"
```

## Data Mapping

### Poison Message Context Preservation

When a message moves to the DLQ, critical context must be preserved to enable effective triage and replay. This table defines the metadata that must travel with the dead-lettered message:

| Field | Purpose | Source | Required for Replay? | Notes |
|---|---|---|---|---|
| original_message_id | Trace back to original message | Broker-assigned | Yes | Used for idempotency dedup and audit trail |
| idempotency_key | Prevent duplicate processing on replay | Application-assigned | Yes | Without this, replay creates duplicates in target ERP |
| error_category | Triage classification (transient/permanent/data_quality) | Consumer error handler | Yes | Determines triage path in decision tree |
| error_message | Root cause description | Exception message | No (helpful) | Truncate to 500 chars to avoid message size limits |
| source_queue | Original queue or topic the message came from | Broker metadata | Yes | Required to route replay back to correct source |
| original_timestamp | When the message was first produced | Producer | Yes | Used to detect aging and retention deadline |
| attempt_count | How many times delivery was attempted | Broker (ApproximateReceiveCount / DeliveryCount) | Yes | Helps determine if retry budget is misconfigured |
| target_erp_system | Which ERP the message was destined for | Application routing metadata | No (helpful) | Enables per-system triage dashboards |
| correlation_id | Links related messages in a multi-step flow | Application-assigned | Conditional | Required for ordered replay of parent/child messages |

### Platform-Specific DLQ Metadata

| Platform | Auto-Captured Metadata | Custom Metadata | Access Pattern |
|---|---|---|---|
| AWS SQS | ApproximateReceiveCount, SentTimestamp, ApproximateFirstReceiveTimestamp | MessageAttributes (up to 10, max 256 KB total) | sqs:ReceiveMessage with AttributeNames=All [src4] |
| Azure Service Bus | DeliveryCount, EnqueuedTimeUtc, DeadLetterReason, DeadLetterErrorDescription | Custom properties (unlimited, within message size limit) | Peek/receive on $deadletterqueue path [src3] |
| Apache Kafka | Offset, partition, timestamp | Headers (key-value byte arrays, no size limit) | Consumer on DLT topic with header deserialization [src5] |
| MuleSoft Anypoint MQ | deliveryCount, destination | Custom properties | Anypoint MQ REST API or REM console [src6] |

## Error Handling & Failure Points

### Common Error Codes That Create Poison Messages

| Code | Meaning | Source System | Triage Action |
|---|---|---|---|
| 400 Bad Request | Payload validation failure — missing/invalid fields | Target ERP API | Data quality fix -> replay |
| 404 Not Found | Referenced record does not exist (e.g., parent customer) | Target ERP API | Create missing parent -> replay children in order |
| 409 Conflict | Duplicate record — already exists in target | Target ERP API | Safe to discard — message was already processed |
| 422 Unprocessable Entity | Business rule violation (e.g., closed fiscal period) | Target ERP API | Fix target ERP state or escalate -> conditional replay |
| INVALID_FIELD | Field does not exist or is not writable | Salesforce API | Schema mismatch — update field mapping -> replay |
| UNABLE_TO_LOCK_ROW | Record locked by concurrent transaction | Salesforce API | Transient — should not reach DLQ; increase retry budget |
| -ERR_PARSE | Malformed XML/JSON — cannot deserialize | Any consumer | Permanent — discard + log for producer fix |
| GOVERNANCE_LIMIT | SuiteScript governance units exhausted | NetSuite | Transient — reduce batch size, replay in smaller chunks |

### Failure Points in Production

- **Silent DLQ message expiration**: AWS SQS DLQ messages retain original enqueue timestamp. A message that spent 10 days in the source queue before DLQ routing has only 4 days of retention left. Fix: `Set DLQ retention period to maximum (14 days) independent of source queue; monitor ApproximateAgeOfOldestMessage metric`. [src4]
- **Replay creates duplicates**: Replayed messages processed successfully but consumer crashed before acknowledging. On retry, message processes again. Fix: `Implement idempotency check in consumer using message_id or business key before write — upsert pattern, not insert`. [src1]
- **Ordered replay violation**: Parent invoice header replayed after child line items, causing FK violations in target ERP. Fix: `Sort replay batch by correlation_id + sequence_number; replay parent records first, wait for confirmation, then replay children`. [src1]
- **DLQ consumer failure creates infinite loop**: DLQ triage consumer crashes and its messages go to a "DLQ of DLQ." Fix: `Never assign a DLQ to a DLQ consumer. Log errors, alert, and halt processing — investigate triage logic`. [src1, src7]
- **Replay storm overwhelms target ERP**: Replaying 50,000 accumulated DLQ messages at full speed triggers rate limiting (429) on the target ERP, creating new failures. Fix: `Use velocity-controlled replay — AWS SQS custom redrive velocity, or application-level throttle at 50-100 messages/second`. [src4]
- **Stale DLQ messages replay into changed schema**: Target ERP schema changed since the message was produced (e.g., new required field added). Fix: `Validate message schema compatibility before replay; transform stale messages to current schema if possible, discard if not`. [src7]

## Anti-Patterns

### Wrong: Infinite retry loop with no DLQ

```python
# BAD — message that fails due to schema violation retries forever
def process_message(message):
    while True:
        try:
            call_erp_api(message)
            return
        except Exception:
            time.sleep(5)  # retry in 5 seconds
            # Never gives up. Blocks queue. Burns compute.
```

### Correct: Bounded retry with classification and DLQ routing

```python
# GOOD — classify error, retry transient only, DLQ permanent failures
def process_message(message, max_retries=5):
    for attempt in range(max_retries):
        try:
            call_erp_api(message)
            return  # success
        except TransientError:
            delay = min(2 ** attempt + random.uniform(0, 1), 60)
            time.sleep(delay)  # exponential backoff with jitter
        except (ValidationError, SchemaError) as e:
            route_to_dlq(message, category="data_quality", error=str(e))
            return  # don't retry — data fix needed
        except Exception as e:
            route_to_dlq(message, category="permanent", error=str(e))
            return
    # Exhausted retries on transient errors
    route_to_dlq(message, category="transient_exhausted", error="max retries")
```

### Wrong: Silent message discard on failure

```python
# BAD — failed messages are logged and forgotten
def process_message(message):
    try:
        call_erp_api(message)
    except Exception as e:
        logger.error(f"Failed to process: {e}")
        # Message is acknowledged and deleted. Data is lost forever.
        acknowledge(message)
```

### Correct: Route to DLQ with full context for later triage

```python
# GOOD — failed messages preserved with diagnostic context
def process_message(message):
    try:
        call_erp_api(message)
    except Exception as e:
        error_context = classify_error(e, message)
        route_to_dlq(
            message,
            category=error_context["category"],
            error=str(e),
            original_timestamp=message.get("sent_timestamp"),
            correlation_id=message.get("correlation_id"),
        )
        # Message acknowledged from source — it is now in the DLQ
        acknowledge(message)
```

### Wrong: Replaying without idempotency check

```python
# BAD — replay sends to ERP without checking if already processed
def replay_from_dlq(dlq_messages):
    for msg in dlq_messages:
        call_erp_api(msg)  # may create duplicate invoice/order
        delete_from_dlq(msg)
```

### Correct: Replay with idempotency verification

```python
# GOOD — check if message was already successfully processed before replay
def replay_from_dlq(dlq_messages):
    for msg in dlq_messages:
        idempotency_key = msg.get("idempotency_key")
        if is_already_processed(idempotency_key):
            delete_from_dlq(msg)  # already handled
            continue
        try:
            call_erp_api_with_upsert(msg)  # upsert, not insert
            mark_as_processed(idempotency_key)
            delete_from_dlq(msg)
        except Exception as e:
            # Failed again — quarantine, do not infinite-loop
            quarantine(msg, reason=str(e))
```

## Common Pitfalls

- **DLQ as permanent storage**: Teams treat the DLQ as an archive and never process messages. DLQ depth grows to thousands. Messages silently expire. Fix: `Implement SLA-based triage — critical streams: <4h first triage, <24h resolution. Monitor DLQ depth and age as production metrics`. [src1]
- **No visibility into DLQ contents**: Operations team knows the DLQ has 500 messages but cannot inspect them without consuming (destructive read). Fix: `Use peek/browse operations (Azure Service Bus peek-lock, AWS SQS ReceiveMessage with visibility timeout, Kafka consumer with manual offset management) before committing to action`. [src3, src4]
- **Replaying at full speed after outage**: After a 12-hour outage, 50K messages accumulated in DLQ. Replaying all at once overwhelms the target ERP with a burst 100x normal traffic. Fix: `Implement velocity-controlled replay — start at 10% of normal throughput and ramp up. AWS SQS Redrive API supports custom max velocity`. [src4]
- **Missing correlation between parent and child messages**: An order header fails and its 15 line items also fail (cascading DLQ). Replaying line items before the header creates orphaned records. Fix: `Tag related messages with correlation_id and sequence_number. Sort replay batches by correlation_id, then sequence_number ascending`. [src1]
- **Treating all DLQ messages the same**: A single triage process handles 400 (bad data), 401 (auth failure), and 503 (transient) identically — manual review for everything. Fix: `Automated classification at DLQ ingestion. Auto-replay transient errors on system recovery. Route data quality errors to auto-fix pipeline. Only escalate truly novel errors to manual review`. [src1, src7]
- **No DLQ in development/staging**: DLQ is configured in production but not in dev/staging. Developers never see poison message behavior until production. Fix: `Mirror DLQ configuration in all environments. Include poison message scenarios in integration test suites`. [src7]

## Diagnostic Commands

```bash
# === AWS SQS DLQ Diagnostics ===

# Check DLQ message count
aws sqs get-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/erp-orders-dlq \
  --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible

# Check oldest message age in DLQ (seconds)
aws sqs get-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/erp-orders-dlq \
  --attribute-names ApproximateAgeOfOldestMessage

# Initiate DLQ redrive to source queue (AWS SQS Redrive API)
aws sqs start-message-move-task \
  --source-arn arn:aws:sqs:us-east-1:123456789:erp-orders-dlq \
  --destination-arn arn:aws:sqs:us-east-1:123456789:erp-orders \
  --max-number-of-messages-per-second 50

# Check redrive task status
aws sqs list-message-move-tasks \
  --source-arn arn:aws:sqs:us-east-1:123456789:erp-orders-dlq

# === Azure Service Bus DLQ Diagnostics ===

# Check DLQ message count
az servicebus queue show \
  --resource-group erp-integration \
  --namespace-name erp-bus \
  --name erp-orders \
  --query "countDetails.deadLetterMessageCount"

# Peek DLQ messages (non-destructive)
az servicebus queue peek \
  --resource-group erp-integration \
  --namespace-name erp-bus \
  --name erp-orders \
  --dead-letter

# === Apache Kafka DLT Diagnostics ===

# Check DLT topic lag (consumer group offset vs latest offset)
kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --describe --group dlq-triage

# Count messages in DLT topic
kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list broker:9092 \
  --topic erp-orders-dlt

# === MuleSoft Anypoint MQ Diagnostics ===

# Check queue stats via Anypoint MQ API
curl -X GET "https://anypoint.mulesoft.com/mq/admin/api/v1/organizations/{orgId}/environments/{envId}/regions/{region}/destinations/erp-orders-dlq/stats" \
  -H "Authorization: Bearer $ANYPOINT_TOKEN"
```

## Version History & Compatibility

| Feature | Release Date | Platform | Breaking Changes | Migration Notes |
|---|---|---|---|---|
| SQS DLQ Redrive API (StartMessageMoveTask) | 2024-06 | AWS SQS | N/A (new feature) | Replaces custom redrive consumers; supports velocity control [src4] |
| Anypoint MQ REM (Resubmit Error Messages) | 2025-01 | MuleSoft | N/A (new feature) | Managed replay with visibility — replaces manual DLQ consume + re-publish [src6] |
| Azure Service Bus MaxDeliveryCount | GA | Azure | N/A | Default 10; recommend reducing to 5 for ERP integrations to avoid wasting 10 attempts on permanent failures [src3] |
| Kafka Spring @RetryableTopic + DLT | 2021 (Spring Kafka 2.7) | Kafka/Spring | N/A | Creates retry-N and -dlt topics automatically; configure maxRetryAttempts and backoff [src5] |
| RabbitMQ Quorum Queue delivery-limit | 2020 (RabbitMQ 3.8) | RabbitMQ | Classic queues don't support delivery-limit | Must migrate classic queues to quorum queues for native DLQ delivery counting |
| Boomi Event Streams DLQ | 2024 | Boomi | N/A | Configurable max retries with exponential backoff (5s, 10s, 20s...); auto-route to DLQ on exhaustion |

### Deprecation Policy

Poison message handling is a stable pattern — the core concept (Enterprise Integration Patterns, 2003) has not changed. Platform implementations add features (like AWS SQS Redrive API) but do not deprecate existing DLQ behavior. Monitor vendor release notes for new triage/replay capabilities that may simplify custom implementations. [src2]

## When to Use / When Not to Use

| Use When | Don't Use When | Use Instead |
|---|---|---|
| Messages repeatedly fail and block queue processing | Simple transient failures that resolve with retry + backoff | business/erp-integration/error-handling-dead-letter-queues/2026 |
| Failed messages must be diagnosed, fixed, and replayed | Fire-and-forget integrations where message loss is acceptable | Simple error logging + monitoring |
| Multi-step integration flows where child messages depend on parent message success | Single API call with synchronous response | Direct API error handling with retry |
| Compliance/audit requirements mandate no data loss in integration pipeline | High-throughput streaming where per-message triage is cost-prohibitive | Batch error aggregation + statistical monitoring |
| Multiple failure categories require different remediation paths | All failures have the same root cause and resolution | Single-path retry strategy |

## Cross-System Comparison

| Capability | AWS SQS | Azure Service Bus | Apache Kafka | MuleSoft Anypoint MQ | Boomi Atom Queue |
|---|---|---|---|---|---|
| DLQ Architecture | Separate standard queue | Sub-queue ($deadletterqueue) | Separate topic (DLT) | Separate standard queue | Built-in DLQ per queue |
| Auto Dead-Letter Routing | Yes (redrive policy) | Yes (MaxDeliveryCount) | No (application-level) | Yes (max delivery attempts) | Yes (after 7 attempts) |
| Max Delivery Count Config | 1-1000 | 1-2000 (default 10) | Application-defined | Configurable | Fixed at 7 |
| Native Replay/Redrive | Yes (Redrive API, velocity control) | No (manual receive + re-send) | No (application-level) | Yes (REM feature) | Yes (resend dead letters) |
| DLQ Retention | Source queue retention (max 14 days) | Unlimited (Premium tier) | Topic retention config | 7 days default | Atom storage lifecycle |
| Dead-Letter Reason Metadata | Custom message attributes | DeadLetterReason + DeadLetterErrorDescription | Custom headers | Custom properties | Limited |
| DLQ Message Inspection | ReceiveMessage (visibility timeout) | Peek-lock (non-destructive peek) | Consumer on DLT topic | Anypoint MQ API | Queue Management panel |
| FIFO/Ordering in DLQ | FIFO DLQ for FIFO source queue | FIFO within sessions | Partition-ordered | FIFO queue support | Not guaranteed |
| DLQ Monitoring | CloudWatch: ApproximateNumberOfMessages, ApproximateAgeOfOldestMessage | Azure Monitor: DeadletteredMessages metric | Consumer group lag on DLT | Anypoint Monitoring | Boomi dashboard |

## Important Caveats

- Poison message handling is downstream of retry strategy — if your retry/backoff configuration is wrong, messages reach the DLQ that should not be there. Review the error-handling-dead-letter-queues card first. [src1]
- AWS SQS does NOT reset the message retention timer when a message moves to the DLQ — messages can silently expire in the DLQ before triage completes. Always set DLQ retention to maximum (14 days) and monitor ApproximateAgeOfOldestMessage. [src4]
- Azure Service Bus DLQ messages persist indefinitely with no automatic cleanup — without a triage consumer or purge job, DLQ storage grows unbounded and may hit entity size limits. [src3]
- Kafka has no native DLQ/DLT mechanism — dead letter topics are an application-level pattern. Spring Kafka @RetryableTopic provides automation, but non-Spring consumers must implement DLT routing manually. [src5]
- Replaying large DLQ backlogs can overwhelm the target ERP with burst traffic far exceeding normal volume. Always implement velocity-controlled replay with configurable throughput limits. [src4]
- This card covers message-level poison handling. For distributed transaction coordination (compensating transactions, saga rollback), see the saga pattern card.

## Related Units

- [Error Handling and Dead Letter Queues for ERP Integrations](/business/erp-integration/error-handling-dead-letter-queues/2026) — prerequisite: retry strategies and DLQ fundamentals
- [Idempotency Patterns for ERP Integrations](/business/erp-integration/idempotency-erp-integrations/2026) — required for safe message replay
- [Saga Pattern for Distributed ERP Transactions](/business/erp-integration/saga-pattern-erp-transactions/2026) — for multi-step transaction coordination
- [Change Data Capture for ERP](/business/erp-integration/change-data-capture-erp/2026) — upstream event source that generates messages
- [Order-to-Cash Integration](/business/erp-integration/order-to-cash-integration/2026) — business process where poison messages commonly occur