---
# === IDENTITY ===
id: business/erp-integration/batch-vs-realtime-integration/2026
canonical_question: "When should you use batch vs real-time vs event-driven integration patterns for ERPs?"
aliases:
  - "batch versus real-time ERP integration decision"
  - "event-driven vs batch vs synchronous integration pattern"
  - "how to choose between batch and real-time ERP data sync"
  - "ERP integration pattern selection criteria"
entity_type: erp_integration
domain: business > erp-integration > batch-vs-realtime-integration
region: global
jurisdiction: global
temporal_scope: 2024-2026

# === SYSTEM PROFILE ===
systems:
  - name: "Cross-Platform Reference"
    vendor: "Multiple (Salesforce, SAP, Oracle, Microsoft, NetSuite, Workday)"
    version: "2025-2026 releases"
    edition: "All editions"
    deployment: "cloud"
    api_surface: "REST, SOAP, OData, Bulk, Event Streaming, CDC, File-Based"

# === VERIFICATION ===
last_verified: 2026-03-02
confidence: 0.90
version: 1.0
first_published: 2026-03-02

# === TEMPORAL VALIDITY ===
temporal_validity:
  status: evolving
  last_breaking_change: null
  next_review: 2026-08-29
  change_sensitivity: moderate

# === CONSTRAINTS ===
constraints:
  - "Real-time (synchronous) integration creates tight coupling — if the target ERP is down, the calling system blocks or fails"
  - "Batch integration introduces inherent data latency (minutes to hours) — not suitable for financial-impact or scarce-resource decisions"
  - "Event-driven integration requires middleware (message broker or event mesh) — adds infrastructure complexity and cost"
  - "Salesforce CDC + Platform Events share a 50,000-delivery/day allocation (base) — batch data loads can exhaust event allocations"
  - "SAP IDoc is batch-only; SAP S/4HANA Cloud restricts RFC/BAPI access — OData and Event Mesh are the cloud-first patterns"
  - "Bulk API rate limits (e.g., Salesforce 15,000 batches/24h) cap maximum batch throughput regardless of record volume"
  - "Event replay windows are finite (Salesforce CDC: 3 days, Platform Events: 72h) — missed events outside the window are lost"

# === SKIP CONDITIONS ===
skip_this_unit_if:
  - condition: "Need specific API rate limits for a single ERP system"
    use_instead: "business/erp-integration/{system}-rest-api-capabilities/2026"
  - condition: "Need CDC implementation details for Salesforce specifically"
    use_instead: "business/erp-integration/change-data-capture-erp/2026"
  - condition: "Need event-driven pattern comparison across ERPs"
    use_instead: "business/erp-integration/erp-event-driven-comparison/2026"

# === AGENT HINTS ===
inputs_needed:
  - key: integration_pattern
    question: "What integration pattern do you need?"
    type: choice
    options:
      - "real-time sync (individual records, <1s latency)"
      - "batch/bulk (scheduled, high volume)"
      - "event-driven (webhook, CDC, platform events)"
      - "file-based (CSV/XML import/export)"
  - key: data_volume
    question: "What's your daily data volume?"
    type: choice
    options:
      - "< 1,000 records/day"
      - "1,000-100,000 records/day"
      - "> 100,000 records/day"
  - key: direction
    question: "What's the data flow direction?"
    type: choice
    options:
      - "inbound (writing to this ERP)"
      - "outbound (reading from this ERP)"
      - "bidirectional sync"

# === DISTRIBUTION ===
canonical_source: "https://knowledgelib.io/business/erp-integration/batch-vs-realtime-integration/2026"
suggested_citation: "Source: knowledgelib.io — AI Knowledge Library (verified 2026-03-02)"

# === RELATED UNITS ===
related_kos:
  depends_on: []
  related_to:
    - id: "business/erp-integration/erp-event-driven-comparison/2026"
      label: "Event-Driven Pattern Comparison Across ERP Systems"
    - id: "business/erp-integration/erp-rate-limits-comparison/2026"
      label: "ERP Rate Limits Comparison"
    - id: "business/erp-integration/erp-bulk-import-comparison/2026"
      label: "ERP Bulk Import Comparison"
  solves:
    - id: "business/erp-integration/change-data-capture-erp/2026"
      label: "Change Data Capture for ERP Integration"
    - id: "business/erp-integration/idempotency-erp-integrations/2026"
      label: "Idempotency in ERP Integrations"
  alternative_to:
    - id: "business/erp-integration/saga-pattern-erp-transactions/2026"
      label: "Saga Pattern for ERP Distributed Transactions"
  often_confused_with:
    - id: "business/erp-integration/error-handling-dead-letter-queues/2026"
      label: "Error Handling and Dead Letter Queues"

# === SOURCES ===
sources:
  - id: src1
    title: "Integration Patterns — Salesforce Architects"
    author: Salesforce
    url: https://architect.salesforce.com/fundamentals/integration-patterns
    type: official_docs
    published: 2025-01-15
    reliability: authoritative
  - id: src2
    title: "Enterprise Integration Patterns: The Complete Guide for Modern Data Architectures"
    author: Tacnode
    url: https://tacnode.io/post/enterprise-integration-patterns
    type: technical_blog
    published: 2025-06-01
    reliability: high
  - id: src3
    title: "Real-Time vs Batch Integration: Key Differences for ERP ROI"
    author: Concentrus
    url: https://concentrus.com/real-time-vs-batch-integration/
    type: technical_blog
    published: 2025-03-10
    reliability: moderate_high
  - id: src4
    title: "Real Time vs Batch Integrations: How to Decide What Needs Live Data"
    author: Elephas
    url: https://elephas.us/resources/integrations-architecture/realtime-vs-batch-integrations.html
    type: technical_blog
    published: 2025-08-01
    reliability: moderate_high
  - id: src5
    title: "Batch Processing vs. Stream Processing: What's the Difference?"
    author: Snowplow
    url: https://snowplow.io/blog/batch-processing-vs-stream-processing
    type: technical_blog
    published: 2025-04-15
    reliability: moderate_high
  - id: src6
    title: "SAP Integration Patterns: IDoc, RFC, OData, Events"
    author: MetalHatsCats
    url: https://metalhatscats.com/sap/interview-prep/sap-integration-idocs-apis-middleware-interview-prep
    type: community_resource
    published: 2025-07-01
    reliability: moderate
  - id: src7
    title: "Enabling Real-Time Responsiveness with Event-Driven Architecture"
    author: MIT Technology Review
    url: https://www.technologyreview.com/2025/10/06/1124323/enabling-real-time-responsiveness-with-event-driven-architecture/
    type: industry_report
    published: 2025-10-06
    reliability: high
---

# Batch vs Real-Time vs Event-Driven Integration Patterns for ERPs

## TL;DR

- **Bottom line**: Use real-time only when a user is actively waiting or a financial/compliance decision depends on live data; use event-driven for near-real-time change propagation with loose coupling; use batch for everything else — it is cheaper, more reliable, and simpler to operate. [src4]
- **Key limit**: Real-time (synchronous) integration has hard latency ceilings — Salesforce Apex callouts timeout at 120s, SAP RFC calls tie up dialog work processes, and all systems throttle API calls per 24h window. [src1]
- **Watch out for**: Defaulting to real-time integration for everything — tight coupling means an ERP outage cascades to every connected system. [src3]
- **Best for**: Architecture decisions at the start of an ERP integration project, middleware selection, and pattern assignment per data flow.
- **Authentication**: Pattern-independent — all three patterns use the same auth flows (OAuth 2.0, JWT, TBA, certificates) as the underlying API surface.

## System Profile

This card is a cross-platform architecture pattern guide, not specific to a single ERP. It covers the three fundamental integration timing patterns — batch (scheduled), real-time (synchronous), and event-driven (asynchronous near-real-time) — and maps them to concrete capabilities across Salesforce, SAP S/4HANA, Oracle ERP Cloud, Microsoft Dynamics 365, NetSuite, and Workday. The decision framework applies universally; the implementation details (API surfaces, rate limits, event mechanisms) vary by system.

For system-specific rate limits and API capabilities, see the dedicated cards for each ERP.

| Property | Value |
|---|---|
| **Scope** | Cross-platform architecture pattern |
| **Systems Covered** | Salesforce, SAP S/4HANA, Oracle ERP Cloud, Dynamics 365, NetSuite, Workday |
| **Pattern Types** | Batch, Real-Time (Synchronous), Event-Driven (Asynchronous) |
| **iPaaS Relevance** | MuleSoft, Boomi, Workato, Celigo, SAP Integration Suite, OIC |
| **Deployment** | Cloud-first (applicable to hybrid/on-premise with middleware) |
| **Status** | GA — all major ERPs support all three patterns |

## API Surfaces & Capabilities

Each ERP system maps its integration surfaces to the three timing patterns differently. This table shows which API surface to use for each pattern, per ERP. [src1, src6]

| ERP System | Real-Time API | Batch/Bulk API | Event-Driven Mechanism | File-Based Import |
|---|---|---|---|---|
| Salesforce | REST API (v62.0), Composite API | Bulk API 2.0 (CSV, 150MB/file) | Platform Events, CDC, Pub/Sub API | Data Loader, Bulk API CSV |
| SAP S/4HANA | OData v4, BAPI/RFC (on-prem only) | IDoc (batch documents) | Event Mesh, Business Events | BTP Integration Suite |
| Oracle ERP Cloud | REST API, SOAP | FBDI (File-Based Data Import), ESS jobs | Business Events, Oracle Integration Cloud | FBDI (CSV/XML upload) |
| Dynamics 365 | Dataverse Web API (OData v4) | Data Management Framework (DMF), Dual Write | Business Events, Dataverse webhooks | DMF packages (CSV/XML) |
| NetSuite | SuiteTalk REST/SOAP, RESTlets | SuiteQL bulk queries, CSV Import | User Event Scripts, SuiteScript workflows | CSV Import, SDF bundles |
| Workday | REST API, SOAP | EIB (Enterprise Interface Builder), RaaS | Business Process Events | EIB (file-based), Custom Reports |

## Rate Limits & Quotas

Rate limits are the primary factor that forces batch processing for high-volume integrations, regardless of business preference for real-time. [src1]

### Per-Pattern Throughput Limits

| ERP System | Real-Time Limit | Batch/Bulk Limit | Event Delivery Limit | Notes |
|---|---|---|---|---|
| Salesforce | 100K API calls/24h (Enterprise), 5M (Unlimited) | 15,000 Bulk API batches/24h, 150MB per file | 50K event deliveries/day (base), 250K publish/hour | CDC + Platform Events share delivery allocation [src1] |
| SAP S/4HANA | Fair-use throttling, dialog work process pool | IDoc: no hard limit, throughput depends on system sizing | Event Mesh: subscription-based, typically 1M events/month | Cloud restricts RFC; OData is primary real-time surface [src6] |
| Oracle ERP Cloud | REST: throttled per tenant, no published hard limit | FBDI: 250MB per file, ESS job queue | Business Events: subscription-based | FBDI is the dominant bulk pattern |
| Dynamics 365 | 6,000 API calls/5min per user, 60K/5min per org | DMF: entity-dependent limits | Webhooks: 500 concurrent subscriptions | Dual Write bypasses some API limits |
| NetSuite | 10 concurrent requests, governance units per script | CSV Import: 25,000 rows/file, SuiteQL: no hard row limit | User Event Scripts: trigger-based | Governance units (not API call counts) are the real constraint |
| Workday | Throttled per tenant (undisclosed limits) | EIB: file size limits vary by integration | Business Process Events: subscription-based | RaaS (Report as a Service) is best for bulk reads |

### Latency Expectations by Pattern

| Pattern | Typical Latency | Best Case | Worst Case | Suitable For |
|---|---|---|---|---|
| Real-time (synchronous) | 200ms-2s | 50ms | 120s (Salesforce timeout) | User-facing transactions, credit checks, inventory locks |
| Event-driven (async) | 1s-60s | 500ms | Minutes (backpressure) | Order status propagation, record change sync, notifications |
| Near-real-time batch | 5-15 min | 1 min (micro-batch) | 1 hour | Dashboard updates, lead scoring, non-critical status sync |
| Scheduled batch | 1-24 hours | 15 min | 24+ hours | Financial reporting, data warehouse loads, reference data |
| File-based import | 30 min-24 hours | 15 min | Days (manual review) | Data migration, initial loads, regulatory reporting |

## Authentication

Authentication is pattern-independent — the same auth flows apply regardless of whether you are making a real-time API call, submitting a bulk job, or subscribing to events. See the system-specific authentication cards for details.

| Pattern | Auth Consideration | Gotcha |
|---|---|---|
| Real-time | Token must be cached and refreshed proactively — latency of token refresh adds to API call time | OAuth token expiry mid-transaction can cause silent failures |
| Batch/Bulk | Service account with elevated permissions — bulk operations often require admin-level access | Session timeout during long-running batch jobs (SAP: dialog timeout, Salesforce: 2h session) |
| Event-driven | Subscription credentials must be long-lived and auto-renewing | Salesforce Pub/Sub API requires gRPC with OAuth; SAP Event Mesh uses OAuth 2.0 client credentials |

### Authentication Gotchas

- Real-time integrations that refresh OAuth tokens inline add 200-500ms per call — cache tokens and refresh asynchronously before expiry. [src1]
- Batch service accounts in Salesforce run under a dedicated Integration User — governor limits are per-transaction, not per-user, so the integration user hits the same limits as any other user. [src1]
- Event subscriptions that use short-lived tokens will silently disconnect when the token expires — implement automatic reconnection with replay ID to avoid data loss. [src1]

## Constraints
<!-- Agents: read this section before recommending any integration approach from this unit.
     These are hard constraints that must not be violated. -->

- **Real-time creates tight coupling**: If the target ERP is slow or unavailable, the calling system blocks or fails. Design circuit breakers and fallback paths for every synchronous integration. [src3, src4]
- **Batch data is always stale**: Batch windows mean consuming systems operate on data that is minutes to hours old. Never use batch for inventory availability, credit decisions, or compliance checks. [src3]
- **Event-driven requires infrastructure**: You need a message broker (Kafka, RabbitMQ), event mesh (SAP Event Mesh), or platform-native events (Salesforce Platform Events). This adds operational cost and failure surfaces. [src2]
- **Shared event allocations**: Salesforce CDC and Platform Events share a daily delivery allocation (50K base). A large batch data load can exhaust event capacity for other integrations. [src1]
- **Event replay has a finite window**: Salesforce Platform Events retain for 72 hours; CDC retains for 3 days. If a subscriber is offline longer than the replay window, events are permanently lost. [src1]
- **Cloud ERPs restrict legacy patterns**: SAP S/4HANA Cloud does not support RFC/BAPI; Oracle ERP Cloud limits SOAP API availability. Cloud-first patterns are OData, REST, and Event Mesh. [src6]
- **Bulk API limits cap throughput**: Salesforce Bulk API 2.0 is capped at 15,000 batches/24h. For very large data migrations, plan multi-day runs or off-hours processing. [src1]

## Integration Pattern Decision Tree

Use this decision tree to assign the correct integration pattern for each data flow in your ERP integration project. [src1, src4]

```
START — Assign integration pattern for a specific data flow
├── Is someone actively waiting for the result?
│   ├── YES — Is it a financial-impact or compliance decision?
│   │   ├── YES → REAL-TIME (synchronous API call)
│   │   │   Examples: credit check, payment authorization, inventory reservation
│   │   │   Implementation: REST API / OData / Composite API
│   │   └── NO — Would a 5-minute delay cause genuine business impact?
│   │       ├── YES → REAL-TIME (synchronous API call)
│   │       │   Examples: customer lookup during phone call, price quote
│   │       └── NO → EVENT-DRIVEN (near-real-time async)
│   │           Examples: order status update, case creation notification
│   └── NO — Is the data flow triggered by a record change?
│       ├── YES — Do multiple downstream systems need this change?
│       │   ├── YES → EVENT-DRIVEN (pub/sub pattern)
│       │   │   Implementation: CDC, Platform Events, Event Mesh, webhooks
│       │   │   Examples: customer update → sync to 5 systems
│       │   └── NO → EVENT-DRIVEN or BATCH (based on volume)
│       │       ├── < 1,000 records/day → EVENT-DRIVEN (simpler, lower latency)
│       │       └── > 1,000 records/day → BATCH (more efficient, less API overhead)
│       └── NO — Is it a scheduled data synchronization?
│           ├── YES — Volume per run?
│           │   ├── < 2,000 records → REST API (batch-style, simpler)
│           │   ├── 2,000-150,000 records → BULK API (single job)
│           │   ├── > 150,000 records → BULK API with job chunking or FILE-BASED
│           │   └── Full data migration → FILE-BASED IMPORT (FBDI, DMF, CSV)
│           └── NO — Is it reference data (products, price lists, org hierarchy)?
│               ├── YES → BATCH (nightly/hourly, reference data changes slowly)
│               └── NO → Analyze case-by-case using criteria above
├── Error tolerance?
│   ├── Zero-loss required → EVENT-DRIVEN with replay + dead letter queue
│   ├── Retry acceptable → EVENT-DRIVEN with at-least-once delivery
│   └── Best-effort → BATCH with reconciliation report
└── Bidirectional sync needed?
    ├── YES → Design conflict resolution strategy FIRST
    │   └── Last-writer-wins, source-of-truth per field, or manual resolution
    └── NO → Proceed with chosen pattern
```

## Quick Reference

### Pattern Comparison Summary

| Criterion | Real-Time (Synchronous) | Event-Driven (Async) | Batch (Scheduled) |
|---|---|---|---|
| **Latency** | 50ms-2s | 1s-60s | Minutes to hours |
| **Coupling** | Tight — both systems must be available | Loose — broker decouples systems | Loose — no runtime dependency |
| **Throughput** | Low (API call per record) | Medium (event stream) | High (bulk processing) |
| **Complexity** | Low (simple API call) | High (broker, replay, ordering) | Medium (scheduling, monitoring) |
| **Error handling** | Immediate — caller gets error response | Deferred — dead letter queue, replay | Deferred — reconciliation reports |
| **Data freshness** | Real-time (current) | Near-real-time (seconds behind) | Stale (minutes to hours behind) |
| **Infrastructure** | API endpoint only | Message broker / event mesh | Job scheduler, bulk API access |
| **Cost (relative)** | High (per-call API charges, always-on) | Medium (broker licensing, compute) | Low (off-peak compute, batched calls) |
| **Reliability** | Low (cascading failures) | High (broker persistence, replay) | High (retry entire batch, reconcile) |
| **Best for** | User-facing, low-volume, critical decisions | Multi-system sync, change propagation | High-volume loads, reporting, migrations |
| **Worst for** | High-volume data loads, background jobs | Simple point-to-point, low frequency | Time-sensitive operations, user-facing |

### Pattern-to-ERP Surface Mapping

| Pattern | Salesforce | SAP S/4HANA | Oracle ERP Cloud | Dynamics 365 | NetSuite |
|---|---|---|---|---|---|
| Real-time | REST/Composite API | OData v4, BAPI (on-prem) | REST API | Dataverse Web API | SuiteTalk REST, RESTlets |
| Event-driven | CDC, Platform Events, Pub/Sub API | Event Mesh, Business Events | Business Events, OIC | Business Events, Webhooks | User Event Scripts |
| Batch/Bulk | Bulk API 2.0 | IDoc, BTP Integration | FBDI, ESS jobs | DMF, Dual Write | CSV Import, SuiteQL |
| File-based | Data Loader CSV | BTP file upload | FBDI (CSV/XML) | DMF packages | CSV Import |

## Step-by-Step Integration Guide

### 1. Audit and categorize all data flows

Enumerate every integration point between your ERP and connected systems. For each, capture: direction (inbound/outbound/bidirectional), volume (records/day), latency requirement, and business criticality. [src4]

```
# Integration audit spreadsheet structure
Flow ID | Source System | Target System | Direction | Records/Day | Latency Need | Business Impact | Pattern
F-001   | Salesforce    | NetSuite      | Outbound  | 500         | < 5 min      | High (revenue)  | Event-Driven
F-002   | Warehouse     | Salesforce    | Inbound   | 50,000      | Nightly OK   | Medium          | Batch
F-003   | Website       | ERP           | Inbound   | 200         | < 1s         | Critical (order)| Real-Time
F-004   | ERP           | Data Lake     | Outbound  | 2,000,000   | Nightly OK   | Low             | File-Based
```

**Verify**: Every integration point has a pattern assignment. No flow should be left as "TBD."

### 2. Apply the decision tree to each flow

Walk each flow through the decision tree above. The four questions are: (1) Is someone waiting? (2) Is it change-triggered? (3) What is the volume? (4) What is the error tolerance? [src4]

```
# Decision criteria for F-003 (Website → ERP order creation)
1. Someone waiting? YES — customer sees order confirmation
2. Financial impact?  YES — payment authorization
3. Volume?           200/day — well within real-time API limits
4. Error tolerance?  Zero-loss — must confirm or reject
→ Pattern: REAL-TIME (synchronous REST API call)
```

**Verify**: Pattern assignment is consistent — no flow should be assigned real-time if volume exceeds 10,000 records/day without explicit justification.

### 3. Design error handling per pattern

Each pattern requires a different error handling strategy. [src2, src3]

```python
# Real-time: Circuit breaker pattern
import time

class CircuitBreaker:
    def __init__(self, failure_threshold=5, recovery_timeout=60):
        self.failure_count = 0
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.last_failure_time = None
        self.state = "CLOSED"  # CLOSED=normal, OPEN=failing, HALF_OPEN=testing

    def call(self, func, *args, **kwargs):
        if self.state == "OPEN":
            if time.time() - self.last_failure_time > self.recovery_timeout:
                self.state = "HALF_OPEN"
            else:
                raise Exception("Circuit breaker OPEN — ERP unavailable")
        try:
            result = func(*args, **kwargs)
            if self.state == "HALF_OPEN":
                self.state = "CLOSED"
                self.failure_count = 0
            return result
        except Exception as e:
            self.failure_count += 1
            self.last_failure_time = time.time()
            if self.failure_count >= self.failure_threshold:
                self.state = "OPEN"
            raise
```

**Verify**: Each pattern has: (1) retry policy, (2) failure escalation path, (3) data recovery mechanism.

### 4. Validate rate limit headroom

Calculate your daily API consumption per pattern and compare against ERP limits. Leave at least 30% headroom for spikes. [src1]

```
# Rate limit calculation example for Salesforce
Flows using REST API:
  F-003: 200 calls/day (order creation)
  F-005: 1,000 calls/day (inventory checks)
  F-007: 500 calls/day (customer lookups)
  Subtotal: 1,700 calls/day

Flows using Bulk API:
  F-002: 2 batch jobs/day (50K records, chunked into 2 jobs)
  F-006: 1 batch job/day (30K records)
  Subtotal: 3 batch jobs/day

Enterprise edition limit: 100,000 REST calls/24h → headroom: 98.3% ✓
Bulk API limit: 15,000 batches/24h → headroom: 99.98% ✓

Event delivery allocation: 50,000/day
  F-001: ~500 CDC events/day
  F-008: ~2,000 platform events/day
  Subtotal: 2,500 events/day → headroom: 95% ✓
```

**Verify**: No integration pattern exceeds 70% of its rate limit ceiling. If it does, move flows from real-time to event-driven or batch.

## Code Examples

### Python: Event-driven integration with retry and dead letter queue

```python
# Input:  Event payload from message broker (Kafka, RabbitMQ, or platform events)
# Output: Processed event or dead-letter routing

import json
import time
import logging
from datetime import datetime, timedelta

logger = logging.getLogger(__name__)

MAX_RETRIES = 5
BACKOFF_BASE = 2  # seconds

def process_event_with_retry(event, target_api_client):
    """Process a single ERP integration event with exponential backoff retry."""
    event_id = event.get("event_id", "unknown")
    retry_count = event.get("retry_count", 0)

    for attempt in range(retry_count, MAX_RETRIES):
        try:
            result = target_api_client.upsert(
                object_type=event["object_type"],
                external_id_field=event["external_id_field"],
                external_id=event["external_id"],
                payload=event["data"]
            )
            logger.info(f"Event {event_id} processed successfully on attempt {attempt + 1}")
            return {"status": "success", "result": result}

        except RateLimitError:
            wait_time = BACKOFF_BASE ** attempt
            logger.warning(f"Rate limited on event {event_id}, waiting {wait_time}s")
            time.sleep(wait_time)

        except TransientError as e:
            wait_time = BACKOFF_BASE ** attempt
            logger.warning(f"Transient error on event {event_id}: {e}, retry in {wait_time}s")
            time.sleep(wait_time)

        except PermanentError as e:
            logger.error(f"Permanent error on event {event_id}: {e}")
            return route_to_dead_letter(event, str(e))

    # Exhausted retries — dead letter
    return route_to_dead_letter(event, f"Exhausted {MAX_RETRIES} retries")


def route_to_dead_letter(event, reason):
    """Route failed event to dead letter queue for manual review."""
    dlq_entry = {
        "original_event": event,
        "failure_reason": reason,
        "failed_at": datetime.utcnow().isoformat(),
        "requires_manual_review": True
    }
    logger.error(f"Dead letter: event {event.get('event_id')} — {reason}")
    # Publish to DLQ topic/queue
    return {"status": "dead_letter", "entry": dlq_entry}
```

### JavaScript/Node.js: Batch integration with chunking and progress tracking

```javascript
// Input:  Array of records to upsert into ERP via Bulk API
// Output: Job status with success/failure counts

// Uses: simple-salesforce-bulk or equivalent bulk API client
const CHUNK_SIZE = 10000; // Records per bulk job
const MAX_CONCURRENT_JOBS = 3;

async function batchUpsert(records, bulkClient, objectType) {
  const chunks = [];
  for (let i = 0; i < records.length; i += CHUNK_SIZE) {
    chunks.push(records.slice(i, i + CHUNK_SIZE));
  }

  console.log(`Processing ${records.length} records in ${chunks.length} chunks`);

  const results = { success: 0, failed: 0, errors: [] };

  // Process chunks with concurrency limit
  for (let i = 0; i < chunks.length; i += MAX_CONCURRENT_JOBS) {
    const batch = chunks.slice(i, i + MAX_CONCURRENT_JOBS);
    const promises = batch.map((chunk, idx) =>
      bulkClient.createJob({
        object: objectType,
        operation: "upsert",
        externalIdFieldName: "External_ID__c",
        data: chunk
      }).then(job => pollJobCompletion(bulkClient, job.id))
    );

    const batchResults = await Promise.allSettled(promises);
    for (const result of batchResults) {
      if (result.status === "fulfilled") {
        results.success += result.value.numberRecordsProcessed;
        results.failed += result.value.numberRecordsFailed;
      } else {
        results.errors.push(result.reason.message);
      }
    }
    console.log(`Progress: ${Math.min((i + MAX_CONCURRENT_JOBS) * CHUNK_SIZE, records.length)}/${records.length}`);
  }
  return results;
}

async function pollJobCompletion(client, jobId, intervalMs = 5000, maxWaitMs = 600000) {
  const start = Date.now();
  while (Date.now() - start < maxWaitMs) {
    const status = await client.getJobStatus(jobId);
    if (status.state === "JobComplete") return status;
    if (status.state === "Failed" || status.state === "Aborted") {
      throw new Error(`Bulk job ${jobId} ${status.state}: ${status.errorMessage}`);
    }
    await new Promise(r => setTimeout(r, intervalMs));
  }
  throw new Error(`Bulk job ${jobId} timed out after ${maxWaitMs / 1000}s`);
}
```

### cURL: Test event subscription (Salesforce Pub/Sub API via CometD)

```bash
# Input:  Salesforce access token, instance URL
# Output: CDC event stream for Account object changes

# Step 1: Subscribe to CDC channel via CometD handshake
curl -X POST "https://YOUR_INSTANCE.salesforce.com/cometd/62.0" \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '[{
    "channel": "/meta/handshake",
    "version": "1.0",
    "supportedConnectionTypes": ["long-polling"]
  }]'

# Step 2: Subscribe to Account change events
curl -X POST "https://YOUR_INSTANCE.salesforce.com/cometd/62.0" \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '[{
    "channel": "/meta/subscribe",
    "clientId": "CLIENT_ID_FROM_HANDSHAKE",
    "subscription": "/data/AccountChangeEvent"
  }]'

# Expected response: subscription confirmation with replayId
# Events will arrive on long-poll connect requests
```

## Data Mapping

### Pattern Selection Matrix by Data Flow Type

| Data Flow Type | Volume/Day | Latency Need | Recommended Pattern | ERP Surface | Error Strategy |
|---|---|---|---|---|---|
| Order creation | 100-10K | < 1s | Real-time | REST/OData API | Circuit breaker + sync error response |
| Order status update | 1K-100K | < 60s | Event-driven | CDC / Platform Events | Replay + dead letter queue |
| Customer master sync | 500-50K | < 15 min | Event-driven or micro-batch | CDC or scheduled REST | Idempotent upsert + reconciliation |
| Product catalog refresh | 10K-500K | Nightly OK | Batch | Bulk API / FBDI | Full reconciliation report |
| Financial posting (GL) | 1K-100K | End of day | Batch | Bulk API / IDoc / FBDI | Zero-loss: validate before commit |
| Inventory snapshot | 10K-1M | Hourly OK | Batch | Bulk query / RaaS | Stale data is expected — timestamp it |
| Price list update | 1K-50K | < 1 hour | Batch or event-driven | REST bulk / CDC | Version-stamp prices, rollback on failure |
| Compliance/audit data | Varies | Real-time | Real-time | REST API (synchronous) | Must succeed — block transaction if fail |

### Data Type Gotchas

- Batch integrations must handle full-vs-delta loads — always prefer delta (changed records only) to avoid reprocessing overhead and rate limit waste. [src1]
- Event-driven integrations receive changes but not the full record — your subscriber may need to call back to the source API for related data, adding latency. [src1]
- Real-time integrations across time zones must normalize timestamps — Salesforce stores UTC, SAP stores in user timezone, NetSuite depends on preference. [src3]

## Error Handling & Failure Points

### Common Error Patterns by Integration Type

| Pattern | Error Type | Frequency | Impact | Resolution |
|---|---|---|---|---|
| Real-time | 429 Rate Limit | Common at scale | Blocked transactions | Exponential backoff; move to batch if persistent |
| Real-time | Timeout (504/408) | Occasional | Hung transactions | Circuit breaker; async fallback |
| Real-time | ERP down (503) | Rare but critical | Cascading failure | Circuit breaker; queue and retry |
| Event-driven | Missed events | Rare | Data inconsistency | Replay from last known ID; periodic reconciliation |
| Event-driven | Duplicate events | Common | Duplicate records | Idempotent receivers (external ID-based upsert) |
| Event-driven | Out-of-order events | Common | Data corruption | Sequence numbers; last-modified-wins logic |
| Batch | Partial failure | Common | Incomplete sync | Per-record error logging; retry failed subset |
| Batch | Job timeout | Occasional | No data synced | Chunk into smaller jobs; extend batch window |
| Batch | Lock contention | Occasional | Failed records | Sort records by parent key; stagger batch starts |

### Failure Points in Production

- **Real-time cascading failure**: ERP response times degrade from 200ms to 30s under load, causing all connected systems to queue up and eventually timeout. Fix: `Implement circuit breaker with 5-failure threshold and 60s recovery window`. [src3]
- **Event allocation exhaustion**: A large batch data load in Salesforce triggers CDC events that consume the entire 50K/day event delivery allocation, silently dropping events for other integrations. Fix: `Disable CDC on objects during bulk loads, or purchase high-volume add-on license`. [src1]
- **Replay window expiry**: An event subscriber goes offline for 4 days (e.g., maintenance window). Salesforce Platform Events only retain for 72 hours. All events in the gap are permanently lost. Fix: `Implement periodic full reconciliation (daily or weekly) independent of event stream; never rely solely on events for data consistency`. [src1]
- **Batch window overrun**: A nightly batch job that normally takes 2 hours grows to 8 hours as data volume increases, running into business hours and competing with real-time API traffic. Fix: `Monitor batch duration trends; implement adaptive chunking; set hard cutoff time with resume-from-checkpoint capability`. [src5]
- **Idempotency failure in event processing**: An event is delivered twice (at-least-once semantics), and the subscriber creates a duplicate record because it uses INSERT instead of UPSERT. Fix: `Always use upsert with external ID as the idempotency key; never use insert for event-driven integrations`. [src2]

## Anti-Patterns

### Wrong: Defaulting to real-time for all integrations

```python
# BAD — synchronous API call for a 50,000-record nightly product catalog sync
for product in all_products:  # 50,000 products
    response = erp_api.update_product(product)  # 1 API call per product
    if response.status_code == 429:
        time.sleep(60)  # Wait and retry — this will take days
# Result: 50,000 API calls, hits rate limit after ~2 hours,
# takes 12+ hours to complete, blocks other integrations
```

### Correct: Use batch/bulk for high-volume scheduled operations

```python
# GOOD — Bulk API for the same 50,000-record sync
import csv
import io

# Chunk into bulk jobs of 10,000 records
for chunk in chunks(all_products, 10000):
    csv_data = io.StringIO()
    writer = csv.DictWriter(csv_data, fieldnames=chunk[0].keys())
    writer.writeheader()
    writer.writerows(chunk)

    job = bulk_client.create_job("Product2", "upsert", "External_ID__c")
    bulk_client.upload_data(job.id, csv_data.getvalue())
    bulk_client.close_job(job.id)
# Result: 5 bulk jobs, completes in minutes, minimal API call consumption
```

### Wrong: Using batch for time-sensitive financial decisions

```python
# BAD — hourly batch check for credit limits
# Between batches, a customer with exceeded credit can place orders
def hourly_credit_sync():
    customers = erp_api.bulk_query("SELECT Id, Credit_Limit__c FROM Account")
    for customer in customers:
        cache.set(f"credit:{customer.id}", customer.credit_limit)

def check_credit(customer_id, order_amount):
    cached_limit = cache.get(f"credit:{customer_id}")  # Could be up to 1 hour stale
    return order_amount <= cached_limit  # WRONG — approves orders against stale data
```

### Correct: Use real-time for financial-impact decisions

```python
# GOOD — real-time credit check at point of order
def check_credit(customer_id, order_amount):
    # Synchronous API call — blocks until ERP responds
    response = erp_api.get(f"/accounts/{customer_id}/credit_status")
    if response.status_code != 200:
        return False  # Fail-closed: deny if ERP is unreachable
    credit = response.json()
    return order_amount <= (credit["limit"] - credit["used"])
```

### Wrong: Event-driven without idempotency

```python
# BAD — INSERT on every event (duplicates on redelivery)
def handle_order_event(event):
    db.execute("INSERT INTO orders (erp_id, amount, status) VALUES (?, ?, ?)",
               event["order_id"], event["amount"], event["status"])
    # If event is delivered twice, you get duplicate orders
```

### Correct: Event-driven with idempotent upsert

```python
# GOOD — UPSERT on external ID (safe for redelivery)
def handle_order_event(event):
    db.execute("""
        INSERT INTO orders (erp_id, amount, status, last_event_timestamp)
        VALUES (?, ?, ?, ?)
        ON CONFLICT (erp_id) DO UPDATE SET
            amount = EXCLUDED.amount,
            status = EXCLUDED.status,
            last_event_timestamp = EXCLUDED.last_event_timestamp
        WHERE orders.last_event_timestamp < EXCLUDED.last_event_timestamp
    """, event["order_id"], event["amount"], event["status"], event["timestamp"])
    # Safe: duplicate events are no-ops; out-of-order events are rejected
```

## Common Pitfalls

- **Treating "real-time" as a default**: Teams request real-time integration for everything because it sounds better. Reality: 80% of integration flows work fine with batch or event-driven, at lower cost and higher reliability. Fix: `Require justification for real-time: "Who is waiting, and what happens if they wait 5 minutes?"`. [src4]
- **Ignoring batch window growth**: Batch jobs that take 30 minutes today will take 3 hours in a year as data volumes grow. Fix: `Monitor batch duration weekly; implement adaptive chunking that scales with volume; set hard cutoff times with checkpointing`. [src5]
- **No reconciliation layer**: Event-driven integrations are treated as the sole source of truth. When events are missed (broker outage, replay window expiry), data silently diverges. Fix: `Implement daily full reconciliation independent of the event stream — compare record counts and checksums between systems`. [src2]
- **Mixing patterns on the same data object**: Using real-time for creates, batch for updates, and events for deletes on the same object — each pattern has different ordering guarantees. Fix: `Use one primary pattern per data object per direction; if you must mix, implement sequence numbers and conflict resolution`. [src1]
- **Undersizing event infrastructure**: Deploying with a single Kafka partition or default Platform Events allocation, then discovering that peak load exceeds capacity. Fix: `Load-test event infrastructure at 3x expected peak volume; provision partitions and allocations based on peak, not average`. [src2]
- **Not accounting for API limit sharing**: Real-time API calls and batch API calls share the same daily quota in most ERPs (e.g., Salesforce REST limit). A batch overrun can starve real-time integrations. Fix: `Reserve 30% of API allocation for real-time; use Bulk API (separate limit pool) for batch where available`. [src1]

## Diagnostic Commands

```bash
# Check Salesforce API usage (remaining daily calls)
curl "https://YOUR_INSTANCE.salesforce.com/services/data/v62.0/limits" \
  -H "Authorization: Bearer $ACCESS_TOKEN" | jq '.DailyApiRequests, .DailyBulkV2QueryJobs, .DailyDeliveredPlatformEvents'

# Check Salesforce Bulk API job status
curl "https://YOUR_INSTANCE.salesforce.com/services/data/v62.0/jobs/ingest/$JOB_ID" \
  -H "Authorization: Bearer $ACCESS_TOKEN" | jq '.state, .numberRecordsProcessed, .numberRecordsFailed'

# SAP S/4HANA: Check OData service availability
curl "https://YOUR_SAP_HOST/sap/opu/odata/sap/API_BUSINESS_PARTNER/\$metadata" \
  -H "Authorization: Bearer $ACCESS_TOKEN" -H "Accept: application/xml"

# Dynamics 365: Check API throttling headers
curl -v "https://YOUR_ORG.api.crm.dynamics.com/api/data/v9.2/accounts?\$top=1" \
  -H "Authorization: Bearer $ACCESS_TOKEN" 2>&1 | grep -i "x-ms-ratelimit"

# NetSuite: Check governance unit consumption (SuiteScript log)
# View in Setup > Scripting > Script Execution Log
# Or via SuiteQL:
curl "https://YOUR_ACCOUNT.suitetalk.api.netsuite.com/services/rest/query/v1/suiteql" \
  -H "Authorization: OAuth ..." \
  -d '{"q": "SELECT id, status, dateCreated FROM scheduledscriptinstance WHERE status = '\''FAILED'\'' ORDER BY dateCreated DESC FETCH FIRST 10 ROWS ONLY"}'
```

## Version History & Compatibility

| Era | Integration Style | Dominant Pattern | Key Technology | Status |
|---|---|---|---|---|
| Pre-2010 | Point-to-point | Batch file transfer | FTP, flat files, IDocs | Legacy — still common in manufacturing |
| 2010-2015 | ESB-centric | SOA / real-time | SOAP, ESB (MuleSoft, TIBCO, WebSphere) | Declining — ESBs seen as bottlenecks [src2] |
| 2015-2020 | API-first | REST API + batch | REST, Bulk API, JSON, OAuth 2.0 | Current standard for most orgs |
| 2020-2025 | Event-driven | Hybrid (event + batch) | Kafka, CDC, Platform Events, Event Mesh | Growing adoption, especially in cloud-native |
| 2025+ | AI-augmented | Intelligent routing | ML-based pattern selection, auto-scaling | Emerging — iPaaS vendors adding AI routing [src7] |

### Deprecation Trends

- SAP is deprecating RFC/BAPI for cloud integrations — OData and Event Mesh are the future. Existing RFC integrations must migrate before S/4HANA Cloud adoption. [src6]
- Salesforce SOAP API remains supported but receives no new features — REST API and Pub/Sub API are the strategic investment areas. [src1]
- File-based imports (FBDI, DMF) are not deprecated but increasingly positioned as "initial load only" — ongoing integrations should use API or event patterns. [src1]

## When to Use / When Not to Use

| Use When | Don't Use When | Use Instead |
|---|---|---|
| Starting a new ERP integration project — need to assign patterns to all data flows | Already know you need a specific API surface (e.g., Bulk API) | System-specific API capability card |
| Evaluating whether to move from batch to real-time for a specific flow | Need implementation code for a specific ERP's event mechanism | business/erp-integration/change-data-capture-erp/2026 |
| Designing a hybrid integration architecture across multiple ERPs | Comparing event-driven capabilities across specific ERP systems | business/erp-integration/erp-event-driven-comparison/2026 |
| Justifying pattern choices to stakeholders or architecture review boards | Need rate limit numbers for a specific ERP edition | business/erp-integration/erp-rate-limits-comparison/2026 |
| Troubleshooting pattern mismatch issues (e.g., real-time integration hitting rate limits) | Need error handling patterns specifically (dead letter queues, saga) | business/erp-integration/error-handling-dead-letter-queues/2026 |

## Cross-System Comparison

| Capability | Salesforce | SAP S/4HANA | Oracle ERP Cloud | Dynamics 365 | NetSuite |
|---|---|---|---|---|---|
| **Real-time API** | REST API v62.0 | OData v4 (Cloud), BAPI/RFC (on-prem) | REST API | Dataverse Web API (OData v4) | SuiteTalk REST + RESTlets |
| **Real-time limit** | 100K-5M calls/24h by edition | Fair-use throttling | Per-tenant throttling | 6K/5min per user | 10 concurrent requests |
| **Bulk/Batch API** | Bulk API 2.0 (150MB/file) | IDoc (batch documents) | FBDI (250MB/file) | DMF (entity packages) | CSV Import (25K rows) |
| **Event mechanism** | CDC + Platform Events + Pub/Sub API | Event Mesh + Business Events | Business Events + OIC | Business Events + Webhooks | User Event Scripts |
| **Event retention** | 72h (Platform Events), 3d (CDC) | Subscription-dependent | Subscription-dependent | Subscription-dependent | N/A (trigger-based) |
| **Event delivery guarantee** | At-least-once with replay | At-least-once (Event Mesh) | At-least-once | At-least-once | N/A |
| **File-based import** | Data Loader CSV | BTP file upload | FBDI (CSV/XML) | DMF packages | CSV Import |
| **Best batch pattern** | Bulk API 2.0 (async, parallel) | IDoc + Process Integration | FBDI + ESS scheduled jobs | DMF + recurring imports | SuiteQL + scheduled scripts |
| **Best event pattern** | CDC for record changes, Platform Events for custom | Event Mesh (cloud-first) | Business Events + OIC subscriptions | Dataverse webhooks | User Event Scripts |
| **Hybrid maturity** | High — all 3 patterns well-supported | High (Cloud), Medium (ECC) | Medium-High | Medium-High | Medium |

## Important Caveats

- Rate limits and event allocations vary dramatically by ERP edition and licensing tier — always verify against your specific contract and current release notes before finalizing pattern assignments. [src1]
- "Real-time" in ERP integration typically means 200ms-2s latency, not sub-millisecond — if your use case truly requires <50ms, consider caching or data virtualization instead of direct API calls. [src2]
- Event-driven patterns provide eventual consistency, not strong consistency — if your business process requires ACID guarantees across systems, use the saga pattern with compensating transactions instead. [src2]
- Cloud ERP editions increasingly restrict legacy integration surfaces (RFC, BAPI, SOAP) — verify that your chosen pattern is supported in your target deployment model before building. [src6]
- The hybrid approach (real-time for critical flows, event-driven for changes, batch for bulk) is the industry consensus for 2025-2026, but it introduces three different error handling strategies, three different monitoring approaches, and three different failure modes to manage. Budget accordingly. [src3, src4]
- Data integration market projected to reach $30.27B by 2030 from $15.18B in 2024 — pattern selection is becoming a strategic capability, not just a technical decision. [src7]

## Related Units

- [Event-Driven Pattern Comparison Across ERP Systems](/business/erp-integration/erp-event-driven-comparison/2026)
- [ERP Rate Limits Comparison](/business/erp-integration/erp-rate-limits-comparison/2026)
- [ERP Bulk Import Comparison](/business/erp-integration/erp-bulk-import-comparison/2026)
- [Change Data Capture for ERP Integration](/business/erp-integration/change-data-capture-erp/2026)
- [Idempotency in ERP Integrations](/business/erp-integration/idempotency-erp-integrations/2026)
- [Saga Pattern for ERP Distributed Transactions](/business/erp-integration/saga-pattern-erp-transactions/2026)
- [Error Handling and Dead Letter Queues](/business/erp-integration/error-handling-dead-letter-queues/2026)