---
# === IDENTITY ===
id: business/erp-integration/bulk-processing-patterns/2026
canonical_question: "What are the best chunking and parallel processing strategies for bulk ERP integration?"
aliases:
  - "ERP bulk data processing chunking strategies across Salesforce SAP NetSuite D365 Oracle"
  - "How to chunk and parallelize large ERP data loads"
  - "Optimal batch sizes and concurrency limits for bulk ERP imports"
  - "Bulk API chunking patterns for enterprise integrations"
entity_type: erp_integration
domain: business > erp-integration > bulk-processing-patterns
region: global
jurisdiction: global
temporal_scope: 2024-2026

# === SYSTEM PROFILE ===
# Cross-system architecture pattern card — covers bulk processing across all major ERPs.
systems:
  - name: "Salesforce"
    vendor: "Salesforce"
    version: "API v62.0"
    edition: "Enterprise, Unlimited"
    deployment: "cloud"
    api_surface: "Bulk API 2.0"
  - name: "Oracle NetSuite"
    vendor: "Oracle"
    version: "2024.2"
    edition: "SuiteCloud"
    deployment: "cloud"
    api_surface: "SuiteTalk REST, CSV Import"
  - name: "SAP S/4HANA"
    vendor: "SAP"
    version: "2408"
    edition: "Cloud, On-Premise"
    deployment: "hybrid"
    api_surface: "OData v4, BAPI, LSMW, LTMC"
  - name: "Microsoft Dynamics 365"
    vendor: "Microsoft"
    version: "10.0.40 (PU64)"
    edition: "Finance & Operations"
    deployment: "cloud"
    api_surface: "OData v4, DMF Recurring Integration"
  - name: "Oracle ERP Cloud"
    vendor: "Oracle"
    version: "26A"
    edition: "Fusion Cloud"
    deployment: "cloud"
    api_surface: "REST, FBDI, ESS"

# === VERIFICATION ===
last_verified: 2026-03-07
confidence: 0.86
version: 1.0
first_published: 2026-03-07

# === TEMPORAL VALIDITY ===
temporal_validity:
  status: volatile
  last_breaking_change: "2025-10 — Salesforce Bulk API 2.0 raised PK chunking default from 100K to 250K records; D365 PU63 introduced Batch Concurrency Control"
  next_review: 2026-09-03
  change_sensitivity: high

# === CONSTRAINTS ===
constraints:
  - "Salesforce Bulk API 2.0: max 150 MB per upload file; 100M records per 24h rolling; 15,000 batches per 24h; max 25 concurrent ingest jobs"
  - "NetSuite REST API: 1 record per create/update request; concurrency 15-55 threads depending on account tier; each SuiteCloud Plus license adds 10 threads"
  - "SAP S/4HANA: BDC batch input limited to 999 records per session; OData batch requests max 1,000 changesets per $batch; RFC/BAPI calls share dialog work process pool"
  - "D365 F&O: sync mode limited to jobs completable within 30 minutes; parallel package import requires Enhanced Parallel Package Import enabled; DMF file-based enqueue shares blob storage with 7-day retention"
  - "Oracle ERP Cloud FBDI: max 250 MB per ZIP file; max 100,000 records per import; max 5 concurrent import activities; individual CSV in ZIP max 1 GB"
  - "Cross-system: chunk sizes too small increase API overhead 5-20x; chunks too large risk timeout, memory exhaustion, and all-or-nothing rollback of entire batch"

# === SKIP CONDITIONS ===
skip_this_unit_if:
  - condition: "User needs real-time individual record sync (< 100 records/operation)"
    use_instead: "business/erp-integration/batch-vs-realtime-integration/2026"
  - condition: "User needs a single ERP system's complete API reference"
    use_instead: "business/erp-integration/{system}-rest-api-capabilities/2026 or {system}-bulk-api-capabilities/2026"
  - condition: "User needs data migration strategy (one-time load, not recurring integration)"
    use_instead: "Data migration planning — use ETL tools with staging tables, not runtime bulk processing patterns"
  - condition: "User needs error handling and retry patterns only"
    use_instead: "business/erp-integration/error-handling-dead-letter-queues/2026"

# === AGENT HINTS ===
inputs_needed:
  - key: erp_system
    question: "Which ERP system(s) are you loading data into?"
    type: choice
    options:
      - "Salesforce"
      - "NetSuite"
      - "SAP S/4HANA"
      - "Dynamics 365 F&O"
      - "Oracle ERP Cloud"
      - "Multiple systems"
  - key: data_volume
    question: "What's your daily data volume?"
    type: choice
    options:
      - "< 10,000 records/day"
      - "10,000-100,000 records/day"
      - "100,000-1,000,000 records/day"
      - "> 1,000,000 records/day"
  - key: direction
    question: "What's the data flow direction?"
    type: choice
    options:
      - "inbound (writing to ERP)"
      - "outbound (reading/extracting from ERP)"
      - "bidirectional sync"

# === DISTRIBUTION ===
canonical_source: "https://knowledgelib.io/business/erp-integration/bulk-processing-patterns/2026"
suggested_citation: "Source: knowledgelib.io — AI Knowledge Library (verified 2026-03-07)"

# === RELATED UNITS ===
related_kos:
  depends_on:
    - id: "business/erp-integration/erp-rate-limits-comparison/2026"
      label: "ERP Rate Limits Comparison — know the limits before designing chunk sizes"
  related_to:
    - id: "business/erp-integration/erp-bulk-import-comparison/2026"
      label: "ERP Bulk Import Comparison — feature-level comparison of bulk APIs"
    - id: "business/erp-integration/batch-vs-realtime-integration/2026"
      label: "Batch vs Real-time Integration — when to use bulk vs event-driven"
    - id: "business/erp-integration/error-handling-dead-letter-queues/2026"
      label: "Error Handling & Dead Letter Queues — handling partial failures in bulk"
  solves:
    - id: "business/erp-integration/salesforce-bulk-api-capabilities/2026"
      label: "Salesforce Bulk API — detailed Salesforce Bulk API 2.0 reference"
    - id: "business/erp-integration/dynamics-365-dmf-import-export/2026"
      label: "D365 DMF Import/Export — Data Management Framework details"
  alternative_to:
    - id: "business/erp-integration/change-data-capture-erp/2026"
      label: "Change Data Capture — event-driven alternative to bulk batch processing"
  often_confused_with:
    - id: "business/erp-integration/idempotency-erp-integrations/2026"
      label: "Idempotency Patterns — complementary but distinct concern from chunking"

# === SOURCES (7 authoritative sources) ===
sources:
  - id: src1
    title: "Bulk API 2.0 and Bulk API Developer Guide — Limits"
    author: Salesforce
    url: https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/bulk_common_limits.htm
    type: official_docs
    published: 2026-02-01
    reliability: authoritative
  - id: src2
    title: "PK Chunking — Bulk API 2.0 Developer Guide"
    author: Salesforce
    url: https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/async_api_headers_enable_pk_chunking.htm
    type: official_docs
    published: 2026-02-01
    reliability: authoritative
  - id: src3
    title: "Optimizing High-Volume NetSuite REST API Integrations"
    author: Houseblend
    url: https://www.houseblend.io/articles/netsuite-rest-api-high-volume-integrations
    type: technical_blog
    published: 2025-08-15
    reliability: high
  - id: src4
    title: "Recurring Integrations — Dynamics 365 Finance & Operations"
    author: Microsoft
    url: https://learn.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/data-entities/recurring-integrations
    type: official_docs
    published: 2026-01-20
    reliability: authoritative
  - id: src5
    title: "Data Import Options and Guidelines for Fusion Applications Suite"
    author: Oracle A-Team
    url: https://www.ateam-oracle.com/data-import-options-and-guidelines-for-fusion-applications-suite
    type: technical_blog
    published: 2025-06-10
    reliability: high
  - id: src6
    title: "Import file-based data to Oracle Fusion Cloud ERP"
    author: Oracle
    url: https://docs.oracle.com/en/solutions/import-data-for-enterprise-resource-planning/index.html
    type: official_docs
    published: 2025-09-01
    reliability: authoritative
  - id: src7
    title: "General Guidelines for Data Loads — Salesforce Bulk API"
    author: Salesforce
    url: https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/asynch_api_planning_guidelines.htm
    type: official_docs
    published: 2026-02-01
    reliability: authoritative
---

# Bulk Processing Patterns for ERP Integration — Chunking & Parallel Processing Strategies

## TL;DR

- **Bottom line**: Break datasets into 1,000-10,000 record chunks (system-dependent), process with rate-limit-aware parallelism, and implement per-chunk error handling — never send everything in one giant batch.
- **Key limit**: Each ERP has hard ceiling limits — Salesforce 150 MB/file and 100M records/24h; NetSuite 15-55 concurrent threads; D365 30-min sync timeout; Oracle FBDI 250 MB/ZIP and 100K records/import. [src1, src3, src4, src6]
- **Watch out for**: Partial success handling — Salesforce Bulk API reports per-record success/failure within a job, but NetSuite CSV import and Oracle FBDI fail entire batches on any validation error. Design your chunking around your ERP's failure granularity.
- **Best for**: Any integration moving > 2,000 records per operation — data migration, nightly syncs, master data distribution, large transaction loads.
- **Authentication**: Use service-to-service OAuth flows (JWT bearer for Salesforce, TBA for NetSuite, Entra ID client credentials for D365, OAuth 2.0 for Oracle) — never tie bulk jobs to human user sessions.

## System Profile

This is a cross-system architecture pattern card covering bulk data processing strategies across the five most widely integrated ERP platforms. It compares native bulk APIs, chunking approaches, concurrency models, and error handling at the bulk operation level. This card does NOT cover real-time individual record operations, event-driven patterns, or one-time data migration tools — see related units for those.

| System | Role | Bulk API Surface | Direction Covered |
|---|---|---|---|
| Salesforce | CRM / Platform | Bulk API 2.0 (CSV over REST) | Inbound + Outbound (PK Chunking) |
| Oracle NetSuite | ERP / Financials | SuiteTalk REST, CSV Import, SuiteQL | Inbound + Outbound |
| SAP S/4HANA | ERP / Manufacturing | OData $batch, BAPI, LSMW, LTMC | Inbound + Outbound |
| Microsoft Dynamics 365 F&O | ERP / Finance | DMF Recurring Integration, OData batch | Inbound + Outbound |
| Oracle ERP Cloud | ERP / Financials | FBDI (File-Based Data Import), ESS, REST | Inbound + Outbound (BICC/BI Publisher) |

## API Surfaces & Capabilities

| ERP System | Bulk API | Protocol | Max Records/Request | Max File Size | Concurrency Limit | Async? | Partial Success? |
|---|---|---|---|---|---|---|---|
| Salesforce | Bulk API 2.0 | REST/CSV | 150M records/job | 150 MB/file | 25 concurrent jobs | Yes | Yes — per-record results |
| NetSuite | CSV Import | UI/Scheduled | 100K+ per file | No hard limit | 5 queues (multi-thread with SuiteCloud Plus) | Yes | No — fails entire batch |
| NetSuite | SuiteTalk REST | REST/JSON | 1 record/request | 104 MB response | 15-55 concurrent (tier-dependent) | Optional | N/A (single record) |
| SAP S/4HANA | OData $batch | REST/JSON | 1,000 changesets/$batch | 50 MB | Shared dialog work processes | Yes (async mode) | Yes — per-changeset |
| SAP S/4HANA | LSMW/LTMC | Batch input | No hard limit (999 per BDC session) | File-dependent | 1 session per project | Yes | Partial — error log |
| D365 F&O | DMF Recurring | REST/Package | Package-dependent | No documented limit | Configurable via Batch Concurrency Control | Yes | Yes — per-record in execution log |
| Oracle ERP Cloud | FBDI | REST/ZIP(CSV) | 100,000 per import | 250 MB/ZIP | 5 concurrent imports | Yes (ESS jobs) | No — fails on validation error |

[src1, src3, src4, src6]

## Rate Limits & Quotas

### Per-Request Limits

| ERP System | Limit Type | Value | Notes |
|---|---|---|---|
| Salesforce | Max file size per upload | 150 MB | Split larger datasets into multiple files |
| Salesforce | Internal batch size | 10,000 records | Auto-created; each takes up to 10 min to process |
| Salesforce | PK Chunking default | 250,000 records | Configurable via Sforce-PK-Chunking header |
| NetSuite | Records per REST request | 1 (create/update) | Use CSV Import for true bulk operations |
| NetSuite | Max response payload | 104 MB | Paginate queries exceeding this |
| SAP | Changesets per $batch | 1,000 | Split larger operations across multiple $batch calls |
| SAP | BDC session limit | 999 records | LSMW has no such limit |
| D365 | Sync mode timeout | 30 minutes | Use batch mode for longer operations |
| Oracle FBDI | Records per import | 100,000 | Split larger datasets into multiple FBDI submissions |
| Oracle FBDI | ZIP file size | 250 MB | Individual CSV within ZIP can be up to 1 GB |

[src1, src2, src4, src6]

### Rolling / Daily Limits

| ERP System | Limit Type | Value | Window | Edition Differences |
|---|---|---|---|---|
| Salesforce | Total records via Bulk API | 100,000,000 | 24h rolling | Same across editions |
| Salesforce | Bulk API batches | 15,000 | 24h rolling | Shared across editions |
| NetSuite | API calls | Tier-dependent | 24h rolling | Tier 1: lower; enterprise accounts: significantly higher |
| NetSuite | Concurrent threads | 15-55 | Per-moment | SuiteCloud Plus adds 10 threads per license |
| SAP | OData requests | Fair-use throttling | Depends on system sizing | On-premise: limited by hardware; Cloud: quota-based |
| D365 | API calls | Priority-based throttling | Per-minute | Based on user license count and tenant config |
| Oracle FBDI | Concurrent imports | 5 | Per-moment | Same across editions |

[src1, src3, src4, src5]

## Authentication

| ERP System | Recommended Flow | Token Lifetime | Notes |
|---|---|---|---|
| Salesforce | OAuth 2.0 JWT Bearer | 2h (session timeout) | New JWT per bulk job; never use username-password |
| NetSuite | Token-Based Auth (TBA) | Until revoked | Create dedicated integration user; OAuth 2.0 also supported |
| SAP S/4HANA | OAuth 2.0 Client Credentials | Configurable (default 12h) | Via SAP BTP or direct; basic auth deprecated |
| D365 F&O | Microsoft Entra ID (OAuth 2.0) | 1h access token | Use client credentials flow; register app in Entra ID |
| Oracle ERP Cloud | OAuth 2.0 | Session-based | Via Oracle Identity Cloud Service (IDCS) |

### Authentication Gotchas

- Salesforce bulk jobs inherit the API user's governor limits — a dedicated integration user prevents limit sharing with interactive users [src1]
- NetSuite TBA tokens are tied to specific integration records — rotating tokens requires updating all connected middleware [src3]
- D365 recurring integration jobs require the Entra ID app ID to be registered in the data project — mismatched app IDs silently reject submissions [src4]

## Constraints
<!-- Agents: read this section before recommending any bulk integration approach. -->

- Salesforce Bulk API 2.0 auto-creates internal 10K-record batches from your uploaded file — you cannot control internal batch boundaries, only the file you upload [src1]
- NetSuite has no native bulk upsert API — CSV Import or individual REST calls with external IDs are the only options for idempotent bulk writes [src3]
- SAP OData $batch is transactional per changeset — a single failing record in a changeset rolls back the entire changeset, not just that record
- D365 DMF parallel package import must be explicitly enabled (Data Management > Framework Parameters > Enhanced Parallel Package Import) — without it, packages process sequentially [src4]
- Oracle FBDI rejects the entire import on any data validation error — there is no partial success mode; pre-validate all data before submission [src6]
- Cross-system: running bulk jobs during business hours competes with interactive users for API quota — schedule bulk operations during off-peak windows

## Integration Pattern Decision Tree

```
START — User needs to bulk-load data into an ERP system
├── How many records per batch cycle?
│   ├── < 2,000 records
│   │   └── Use standard REST API with composite/batch requests (no bulk API needed)
│   ├── 2,000 – 100,000 records
│   │   ├── Salesforce → Bulk API 2.0 single job (auto-chunks at 10K internally)
│   │   ├── NetSuite → CSV Import (enable multi-threading with SuiteCloud Plus)
│   │   ├── SAP → OData $batch (1,000 changesets/request, multiple $batch calls)
│   │   ├── D365 → DMF Recurring Integration (single data package)
│   │   └── Oracle → FBDI single ZIP upload + ESS job
│   ├── 100,000 – 1,000,000 records
│   │   ├── Salesforce → Bulk API 2.0 with file chunking (split into 10-15 MB files)
│   │   ├── NetSuite → CSV Import with queue distribution (queues 1-5)
│   │   ├── SAP → LTMC with migration objects or BAPI batch processing
│   │   ├── D365 → DMF with Enhanced Parallel Package Import
│   │   └── Oracle → Multiple FBDI submissions (≤100K per import, ≤5 concurrent)
│   └── > 1,000,000 records
│       ├── Salesforce → Bulk API 2.0 with job queuing (max 25 concurrent)
│       ├── NetSuite → Staged CSV Import batches + message queue orchestration
│       ├── SAP → LSMW / LTMC with parallel sessions
│       ├── D365 → Multiple DMF packages with batch concurrency control
│       └── Oracle → Sequential FBDI batches (respect 5-concurrent limit)
├── Chunking strategy?
│   ├── Fixed-size chunks → Simplest; use when records are independent
│   ├── Adaptive chunks → Adjust size based on API response times and error rates
│   └── Dependency-aware chunks → Group parent/child records together (e.g., Account + Contacts)
├── Error tolerance?
│   ├── Zero-loss → Implement per-chunk retry + dead letter queue + idempotency keys
│   └── Best-effort → Log failures, skip, continue processing remaining chunks
└── Performance priority?
    ├── Throughput → Maximize parallelism up to concurrency limit
    └── Reliability → Reduce parallelism, increase retry budget per chunk
```

## Quick Reference

### Cross-System Bulk Processing Comparison

| Capability | Salesforce | NetSuite | SAP S/4HANA | D365 F&O | Oracle ERP Cloud |
|---|---|---|---|---|---|
| **Primary Bulk API** | Bulk API 2.0 | CSV Import | LTMC / OData $batch | DMF Recurring Integration | FBDI + ESS |
| **Max records/job** | 150M/file | 100K+ (practical) | 999/BDC session; unlimited LSMW | Package-dependent | 100,000/import |
| **Max file size** | 150 MB | No hard limit | File-dependent | No documented limit | 250 MB ZIP |
| **Max concurrent jobs** | 25 | 5 queues | Session-limited | Configurable (Batch Concurrency Control) | 5 imports |
| **Optimal chunk size** | 10K-50K records/file | 5K-25K records/CSV | 500-1,000 changesets/$batch | 10K-50K records/package | 25K-50K records/FBDI |
| **Partial success** | Yes (per-record) | No (entire batch) | Yes (per-changeset) | Yes (per-record in log) | No (entire import) |
| **Built-in retry** | Yes (10 retries/batch) | No | No | Yes (PU64+ auto-retry) | No |
| **PK Chunking (query)** | Yes (250K default) | N/A | N/A | N/A | N/A |
| **Typical throughput** | ~3,000-5,000 rec/sec | ~3 rec/sec (REST); ~500 rec/sec (CSV) | ~200-1,000 rec/sec (OData) | ~500-2,000 rec/sec | ~300-1,000 rec/sec |
| **Async processing** | Yes | Yes (CSV Import) | Yes (async OData, background jobs) | Yes (batch mode) | Yes (ESS) |

[src1, src2, src3, src4, src5, src6, src7]

## Step-by-Step Integration Guide

### 1. Assess data volume and select chunking strategy

Determine your total record count, record dependencies (parent/child relationships), and target ERP's limits. Choose one of three chunking strategies: [src7]

- **Fixed-size chunks**: Split into equal-sized segments (e.g., 10,000 records per chunk). Best when records are independent.
- **Adaptive chunks**: Start with a baseline size, then adjust based on processing time and error rates. Reduce chunk size if error rate exceeds 5%.
- **Dependency-aware chunks**: Group parent records with their children (e.g., Account + Contacts, Sales Order + Line Items). Prevents referential integrity failures.

**Verify**: Count total records, divide by optimal chunk size for target ERP (see Quick Reference table) → expected: chunk count = ceil(total_records / chunk_size)

### 2. Implement chunk generation with dependency awareness

Split your source data into chunks that respect parent-child relationships. [src7]

```python
# Input:  List of records with parent_id relationships
# Output: List of chunks, each containing complete parent-child groups

def chunk_with_dependencies(records, chunk_size=10000):
    """Split records into chunks preserving parent-child groups."""
    # Group children under their parents
    groups = {}
    orphans = []
    for record in records:
        parent_id = record.get('parent_id')
        if parent_id:
            groups.setdefault(parent_id, []).append(record)
        else:
            groups.setdefault(record['id'], [])
            groups[record['id']].insert(0, record)

    # Pack groups into chunks without splitting families
    chunks = []
    current_chunk = []
    current_size = 0
    for group_id, group_records in groups.items():
        group_size = len(group_records)
        if current_size + group_size > chunk_size and current_chunk:
            chunks.append(current_chunk)
            current_chunk = []
            current_size = 0
        current_chunk.extend(group_records)
        current_size += group_size
    if current_chunk:
        chunks.append(current_chunk)
    return chunks
```

**Verify**: `len(chunks)` → expected: all parent-child groups intact within single chunks

### 3. Implement rate-limit-aware parallel processing

Process chunks in parallel while respecting the target ERP's concurrency limits. [src1, src3]

```python
# Input:  List of data chunks + ERP connection details
# Output: Per-chunk success/failure results

import asyncio
import aiohttp
from asyncio import Semaphore

# Concurrency limits per ERP (conservative defaults)
ERP_CONCURRENCY = {
    'salesforce': 10,     # Max 25 jobs, leave headroom
    'netsuite': 10,       # Depends on account tier (15-55 threads)
    'sap': 5,             # Shared dialog work processes
    'd365': 8,            # Configurable via Batch Concurrency Control
    'oracle_fbdi': 3,     # Max 5 concurrent imports, leave headroom
}

async def process_chunks_parallel(chunks, erp_type, process_fn):
    """Process chunks with rate-limit-aware parallelism."""
    semaphore = Semaphore(ERP_CONCURRENCY.get(erp_type, 5))
    results = []

    async def process_with_limit(chunk_idx, chunk):
        async with semaphore:
            try:
                result = await process_fn(chunk)
                return {'chunk': chunk_idx, 'status': 'success', 'result': result}
            except RateLimitError as e:
                await asyncio.sleep(e.retry_after or 2 ** chunk_idx)
                result = await process_fn(chunk)  # Single retry
                return {'chunk': chunk_idx, 'status': 'success', 'result': result}
            except Exception as e:
                return {'chunk': chunk_idx, 'status': 'failed', 'error': str(e)}

    tasks = [process_with_limit(i, chunk) for i, chunk in enumerate(chunks)]
    results = await asyncio.gather(*tasks)
    return results
```

**Verify**: All chunks processed; failed chunks logged → expected: `sum(1 for r in results if r['status'] == 'success')` equals total chunks minus any irrecoverable failures

### 4. Handle partial failures per ERP

Each ERP handles partial success differently — your error handling must match. [src1, src4, src6]

```python
# Input:  Bulk job results from ERP
# Output: Failed records ready for retry

def handle_partial_failures(results, erp_type):
    """Extract failed records for retry based on ERP's failure model."""

    if erp_type == 'salesforce':
        # Salesforce Bulk API 2.0: per-record success/failure
        failed = [r for r in results if r['sf__Error']]
        return failed  # Retry individual failed records

    elif erp_type == 'netsuite_csv':
        # NetSuite CSV Import: entire batch fails on any error
        if results.get('status') == 'FAILED':
            return results['all_records']  # Retry entire batch after fixing data

    elif erp_type == 'sap_odata':
        # SAP OData $batch: per-changeset success/failure
        failed_changesets = [cs for cs in results['changesets']
                            if cs['status'] >= 400]
        return failed_changesets  # Retry failed changesets only

    elif erp_type == 'd365':
        # D365 DMF: check execution errors via API
        # POST /data/DataManagementDefinitionGroups/
        #   Microsoft.Dynamics.DataEntities.GetExecutionErrors
        errors = results.get('execution_errors', [])
        return errors  # Extract failed record IDs and retry

    elif erp_type == 'oracle_fbdi':
        # Oracle FBDI: entire import fails on validation error
        if results.get('status') == 'ERROR':
            return results['all_records']  # Fix data and resubmit entire batch
```

**Verify**: Failed records extracted → expected: retry queue populated only with recoverable failures

## Code Examples

### Python: Salesforce Bulk API 2.0 Job with Chunking

```python
# Input:  Large CSV file (e.g., 500K contact records)
# Output: Bulk job IDs with success/failure counts per chunk

import requests
import time
import csv
import io

class SalesforceBulkChunker:
    """Chunked Salesforce Bulk API 2.0 upload with rate-limit awareness."""

    MAX_FILE_SIZE = 100 * 1024 * 1024  # 100 MB (leave 50 MB headroom)
    MAX_CONCURRENT_JOBS = 10            # Leave 15 slots for other processes
    POLL_INTERVAL = 10                  # seconds

    def __init__(self, instance_url, access_token):
        self.base_url = f"{instance_url}/services/data/v62.0"
        self.headers = {
            'Authorization': f'Bearer {access_token}',
            'Content-Type': 'text/csv',
        }

    def chunk_csv_file(self, filepath, target_size_mb=100):
        """Split CSV into chunks under target_size_mb."""
        chunks = []
        with open(filepath, 'r') as f:
            reader = csv.reader(f)
            header = next(reader)
            current_chunk = [header]
            current_size = 0
            for row in reader:
                row_size = len(','.join(row).encode('utf-8'))
                if current_size + row_size > target_size_mb * 1024 * 1024:
                    chunks.append(current_chunk)
                    current_chunk = [header]
                    current_size = 0
                current_chunk.append(row)
                current_size += row_size
            if len(current_chunk) > 1:
                chunks.append(current_chunk)
        return chunks

    def submit_job(self, sobject, operation, csv_data):
        """Create and upload a Bulk API 2.0 job."""
        # Create job
        job_resp = requests.post(
            f"{self.base_url}/jobs/ingest",
            headers={**self.headers, 'Content-Type': 'application/json'},
            json={'object': sobject, 'operation': operation,
                  'contentType': 'CSV', 'lineEnding': 'LF'}
        )
        job_id = job_resp.json()['id']
        # Upload data
        buf = io.StringIO()
        writer = csv.writer(buf)
        writer.writerows(csv_data)
        requests.put(
            f"{self.base_url}/jobs/ingest/{job_id}/batches",
            headers=self.headers,
            data=buf.getvalue().encode('utf-8')
        )
        # Close job to start processing
        requests.patch(
            f"{self.base_url}/jobs/ingest/{job_id}",
            headers={**self.headers, 'Content-Type': 'application/json'},
            json={'state': 'UploadComplete'}
        )
        return job_id

    def poll_job(self, job_id):
        """Poll until job completes."""
        while True:
            resp = requests.get(
                f"{self.base_url}/jobs/ingest/{job_id}",
                headers={**self.headers, 'Content-Type': 'application/json'}
            )
            state = resp.json()['state']
            if state in ('JobComplete', 'Failed', 'Aborted'):
                return resp.json()
            time.sleep(self.POLL_INTERVAL)
```

[src1, src7]

### Python: NetSuite CSV Import with Queue Distribution

```python
# Input:  Large dataset to import into NetSuite
# Output: Import job status per queue

import csv
import os

def prepare_netsuite_csv_chunks(records, chunk_size=10000):
    """Split records into NetSuite-compatible CSV chunks."""
    chunks = []
    for i in range(0, len(records), chunk_size):
        chunk = records[i:i + chunk_size]
        chunks.append(chunk)
    return chunks

# NetSuite CSV Import best practices:
# 1. Enable multi-threading (requires SuiteCloud Plus license)
# 2. Distribute imports across queues 1-5
# 3. Ensure rows do NOT need sequential processing
# 4. Clean data: remove BOM, extra whitespace, special chars
# 5. Map fields before import to prevent misalignment

# For programmatic import via SuiteScript:
# Use N/task module to create CSV import tasks
# suitescript_example = """
# define(['N/task'], function(task) {
#     var csvTask = task.create({ taskType: task.TaskType.CSV_IMPORT });
#     csvTask.mappingId = 'CUSTIMPORT_MY_MAPPING';
#     csvTask.importFile = csvFileId;
#     csvTask.name = 'Bulk Import Chunk ' + chunkIndex;
#     var taskId = csvTask.submit();
#     // Monitor via task.checkStatus({ taskId: taskId })
# });
# """
```

[src3]

### Python: Adaptive Chunking with Rate-Limit Backoff

```python
# Input:  Records to process, target ERP API
# Output: All records processed with adaptive chunk sizing

import time
import logging

class AdaptiveChunker:
    """Adjusts chunk size based on API response metrics."""

    def __init__(self, initial_chunk_size=5000, min_chunk=500, max_chunk=50000):
        self.chunk_size = initial_chunk_size
        self.min_chunk = min_chunk
        self.max_chunk = max_chunk
        self.error_count = 0
        self.success_count = 0

    def adjust_chunk_size(self, response_time_ms, had_errors):
        """Adapt chunk size based on last operation's performance."""
        if had_errors:
            self.error_count += 1
            # Reduce chunk size by 50% on errors
            self.chunk_size = max(self.min_chunk, self.chunk_size // 2)
            logging.warning(f"Errors detected, reducing chunk to {self.chunk_size}")
        elif response_time_ms > 30000:  # > 30 seconds
            # Reduce by 25% if response too slow
            self.chunk_size = max(self.min_chunk, int(self.chunk_size * 0.75))
        elif response_time_ms < 5000 and not had_errors:
            # Increase by 25% if fast and error-free
            self.success_count += 1
            if self.success_count >= 3:  # 3 consecutive successes
                self.chunk_size = min(self.max_chunk, int(self.chunk_size * 1.25))
                self.success_count = 0
        return self.chunk_size

    def process_with_backoff(self, records, process_fn):
        """Process all records with adaptive chunking and backoff."""
        offset = 0
        results = []
        retry_wait = 1

        while offset < len(records):
            chunk = records[offset:offset + self.chunk_size]
            start = time.time()
            try:
                result = process_fn(chunk)
                elapsed_ms = (time.time() - start) * 1000
                self.adjust_chunk_size(elapsed_ms, had_errors=False)
                results.append({'offset': offset, 'count': len(chunk),
                                'status': 'success'})
                offset += len(chunk)
                retry_wait = 1  # Reset backoff on success
            except RateLimitError:
                # Exponential backoff: 1s, 2s, 4s, 8s, max 60s
                logging.warning(f"Rate limited, waiting {retry_wait}s")
                time.sleep(retry_wait)
                retry_wait = min(60, retry_wait * 2)
                self.adjust_chunk_size(0, had_errors=True)
                # Do NOT advance offset — retry same chunk
            except Exception as e:
                elapsed_ms = (time.time() - start) * 1000
                self.adjust_chunk_size(elapsed_ms, had_errors=True)
                results.append({'offset': offset, 'count': len(chunk),
                                'status': 'failed', 'error': str(e)})
                offset += len(chunk)  # Skip failed chunk, log for retry
        return results
```

### cURL: Salesforce Bulk API 2.0 Job Lifecycle

```bash
# Input:  Salesforce access token, CSV file
# Output: Job ID, processing status, success/failure counts

# Step 1: Create ingest job
curl -X POST https://yourinstance.salesforce.com/services/data/v62.0/jobs/ingest \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"object":"Contact","operation":"upsert","externalIdFieldName":"External_ID__c","contentType":"CSV","lineEnding":"LF"}'

# Step 2: Upload CSV data (chunk)
curl -X PUT https://yourinstance.salesforce.com/services/data/v62.0/jobs/ingest/JOB_ID/batches \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: text/csv" \
  --data-binary @contacts_chunk_001.csv

# Step 3: Close job to begin processing
curl -X PATCH https://yourinstance.salesforce.com/services/data/v62.0/jobs/ingest/JOB_ID \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"state":"UploadComplete"}'

# Step 4: Poll job status
curl https://yourinstance.salesforce.com/services/data/v62.0/jobs/ingest/JOB_ID \
  -H "Authorization: Bearer $ACCESS_TOKEN"
# Response includes: numberRecordsProcessed, numberRecordsFailed

# Step 5: Get failed records for retry
curl https://yourinstance.salesforce.com/services/data/v62.0/jobs/ingest/JOB_ID/failedResults \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Accept: text/csv"
```

[src1, src7]

## Data Mapping

### Chunking Parameter Reference

| Parameter | Salesforce | NetSuite | SAP S/4HANA | D365 F&O | Oracle FBDI |
|---|---|---|---|---|---|
| **Optimal chunk size (inbound)** | 10K-50K records | 5K-25K records | 500-1,000 changesets | 10K-50K records | 25K-50K records |
| **Optimal file size** | 10-100 MB | < 50 MB recommended | N/A | Package-dependent | < 100 MB per CSV |
| **Compression** | Not supported (CSV only) | N/A | N/A | ZIP packages | DEFLATE compression in ZIP |
| **Character encoding** | UTF-8 | UTF-8, remove BOM | UTF-8 | UTF-8 | UTF-8 |
| **Line ending** | LF recommended | CRLF or LF | N/A | N/A | LF or CRLF |
| **Date format** | ISO 8601 | MM/DD/YYYY or ISO | ISO 8601 | ISO 8601 | YYYY/MM/DD HH:mm:ss |

### Data Type Gotchas

- Salesforce Bulk API 2.0 CSV requires all lookup fields as 18-char IDs, not 15-char — 15-char IDs cause MALFORMED_ID errors in bulk mode [src1]
- NetSuite CSV Import treats empty strings and "null" differently — empty string clears a field, omitting a column preserves existing value [src3]
- SAP OData $batch with JSON payloads requires Edm.Decimal values as strings (e.g., `"12345.67"`), not JSON numbers — numeric precision loss causes rounding errors
- D365 DMF date fields require ISO 8601 format — locale-specific formats (MM/DD/YYYY) are rejected in REST package submission [src4]
- Oracle FBDI date columns require `YYYY/MM/DD HH:mm:ss` format — ISO 8601 with 'T' separator is rejected [src6]

## Error Handling & Failure Points

### Common Error Codes

| ERP | Code | Meaning | Cause | Resolution |
|---|---|---|---|---|
| Salesforce | `UNABLE_TO_LOCK_ROW` | Record locked | Concurrent update to same record | Reduce parallelism; add jitter between chunks touching same records |
| Salesforce | `REQUEST_LIMIT_EXCEEDED` | Daily API limit hit | Too many bulk jobs in 24h | Consolidate into fewer, larger jobs (up to 150 MB) |
| NetSuite | `429 Too Many Requests` | Rate limit exceeded | Burst or daily limit hit | Exponential backoff; honor Retry-After header |
| NetSuite | `CONCURRENT_REQUEST_LIMIT_EXCEEDED` | Thread limit hit | Too many parallel requests | Reduce semaphore count; check account tier |
| SAP | `501 Not Implemented` | $batch not supported | Endpoint doesn't support batch | Use individual requests or BAPI batch |
| D365 | `ProcessedWithErrors` | Partial execution failure | Data validation errors on some records | Call GetExecutionErrors API for details |
| Oracle | `ESS job FAILED` | Import validation error | Data format or constraint violation | Download error report from ESS; fix data and resubmit |

[src1, src3, src4, src6]

### Failure Points in Production

- **Salesforce Bulk API silent timeout**: Bulk API batches can timeout after 10 minutes per internal 10K-record batch — if a batch times out, Salesforce retries up to 10 times silently, then fails the entire job. Fix: `Keep records per file under 100K to avoid cascading timeouts`. [src1]
- **NetSuite CSV Import order dependency**: If your CSV contains both parent and child records, children may process before parents in multi-threaded mode, causing referential integrity failures. Fix: `Upload parent records in a separate CSV first, then children — or disable multi-threading for dependent data`. [src3]
- **D365 DMF parallel execution collision**: When recurring integration files arrive faster than processing, DMF tries to process in parallel, causing row-level lock contention on the same records. Fix: `Enable sequential processing of messages in the recurring job configuration`. [src4]
- **Oracle FBDI BOM character**: UTF-8 BOM (byte order mark) at the start of CSV files causes FBDI to misread the first column header, silently mapping data to wrong fields. Fix: `Save CSV as UTF-8 without BOM; validate with hex editor`. [src6]
- **Cross-system: timezone-dependent batch windows**: Bulk jobs scheduled at "midnight" may overlap with maintenance windows in different availability zones. Fix: `Use UTC timestamps for all scheduling; check ERP maintenance calendar`.

## Anti-Patterns

### Wrong: Processing all records in a single giant batch

```python
# BAD — single-threaded, no chunking, all-or-nothing failure
def bulk_import_bad(records, api_client):
    # Send ALL 500K records in one request
    result = api_client.bulk_upload(records)  # Times out at 150K+ records
    if result.failed:
        # Entire 500K batch must be retried
        bulk_import_bad(records, api_client)  # Infinite retry loop
```

### Correct: Chunked processing with per-chunk error isolation

```python
# GOOD — chunked, parallel, per-chunk error handling
def bulk_import_good(records, api_client, chunk_size=10000):
    chunks = [records[i:i+chunk_size] for i in range(0, len(records), chunk_size)]
    failed_chunks = []
    for i, chunk in enumerate(chunks):
        try:
            result = api_client.bulk_upload(chunk)
            log_success(i, len(chunk), result.processed)
        except Exception as e:
            failed_chunks.append((i, chunk, str(e)))
            log_failure(i, len(chunk), e)
    # Retry only failed chunks (not the entire dataset)
    retry_failed_chunks(failed_chunks, api_client)
```

### Wrong: Ignoring partial success results

```python
# BAD — assumes entire job succeeded or failed
def check_results_bad(job_id, api_client):
    job = api_client.get_job_status(job_id)
    if job.state == 'JobComplete':
        print("All done!")  # WRONG — JobComplete can still have failed records
```

### Correct: Checking per-record success/failure

```python
# GOOD — inspect individual record results
def check_results_good(job_id, api_client):
    job = api_client.get_job_status(job_id)
    if job.state == 'JobComplete':
        if job.number_records_failed > 0:
            # Download and parse failed records CSV
            failures = api_client.get_failed_results(job_id)
            for record in failures:
                dead_letter_queue.add(record)  # Route to DLQ for manual review
            log.warning(f"{job.number_records_failed} records failed out of "
                        f"{job.number_records_processed}")
        log.info(f"{job.number_records_processed - job.number_records_failed} "
                 f"records succeeded")
```

### Wrong: Maximum parallelism without rate-limit awareness

```python
# BAD — fire all chunks simultaneously
import asyncio
async def blast_all_bad(chunks, api_client):
    tasks = [api_client.upload(chunk) for chunk in chunks]
    await asyncio.gather(*tasks)  # 500 concurrent requests → instant 429
```

### Correct: Semaphore-controlled parallelism with backoff

```python
# GOOD — respect concurrency limits
import asyncio
async def controlled_parallel_good(chunks, api_client, max_concurrent=10):
    sem = asyncio.Semaphore(max_concurrent)
    async def upload_with_limit(chunk):
        async with sem:
            for attempt in range(3):
                try:
                    return await api_client.upload(chunk)
                except RateLimitError:
                    await asyncio.sleep(2 ** attempt)
            raise MaxRetriesExceeded()
    await asyncio.gather(*[upload_with_limit(c) for c in chunks])
```

## Common Pitfalls

- **Chunks too small (< 100 records)**: API overhead per request dominates — a 100-record chunk has the same HTTP overhead as a 10,000-record chunk but processes 100x fewer records. Fix: `Start at the ERP's recommended minimum (see Quick Reference) and increase until you hit timeout or memory limits`. [src7]
- **Chunks too large (pushing limits)**: Large chunks risk timeout, memory exhaustion, and all-or-nothing rollback. Fix: `Stay at 50-70% of the ERP's documented maximum to leave headroom for processing overhead and retries`. [src1]
- **Not pre-validating data**: Oracle FBDI and NetSuite CSV Import fail entire batches on validation errors — one bad record kills the entire chunk. Fix: `Validate all records against the ERP's required fields, data types, and picklist values BEFORE submission`. [src6]
- **Running bulk jobs during business hours**: Bulk operations compete with interactive users for API quota, causing both to degrade. Fix: `Schedule bulk operations during off-peak hours; use ERP-specific scheduling (SAP background jobs, D365 batch windows)`.
- **No progress tracking**: When a 50-chunk job fails at chunk 37, you need to know which chunks succeeded to avoid reprocessing. Fix: `Log chunk boundaries (start offset, count, job ID) in a persistent store; implement checkpoint/restart capability`. [src7]
- **Single-threaded bulk processing**: Processing 1M records in a single thread takes hours when the ERP supports parallel jobs. Fix: `Use concurrent processing up to the ERP's concurrency limit (see Quick Reference table) — typical speedup is 3-8x`. [src3]
- **Mixing create and update in same bulk job**: Some ERPs (Salesforce) require upsert with external ID; others (NetSuite CSV Import) have separate import modes. Fix: `Use upsert operations where available; otherwise separate creates and updates into distinct jobs`. [src1]

## Diagnostic Commands

```bash
# === Salesforce: Check Bulk API job status ===
curl -s "https://$SF_INSTANCE/services/data/v62.0/jobs/ingest/$JOB_ID" \
  -H "Authorization: Bearer $SF_TOKEN" | jq '{state, numberRecordsProcessed, numberRecordsFailed}'

# === Salesforce: Check remaining API limits ===
curl -s "https://$SF_INSTANCE/services/data/v62.0/limits" \
  -H "Authorization: Bearer $SF_TOKEN" | jq '{DailyApiRequests, DailyBulkV2QueryJobs, DailyBulkV2QueryFileStorageMB}'

# === NetSuite: Check API usage (via SuiteQL) ===
curl -s "https://$NS_ACCOUNT.suitetalk.api.netsuite.com/services/rest/query/v1/suiteql" \
  -H "Authorization: OAuth ..." \
  -H "Content-Type: application/json" \
  -d '{"q": "SELECT * FROM ApiUsage WHERE dateCreated > SYSDATE - 1"}'

# === D365: Check recurring integration message status ===
curl -X POST "https://$D365_INSTANCE/data/DataManagementDefinitionGroups/Microsoft.Dynamics.DataEntities.GetMessageStatus" \
  -H "Authorization: Bearer $D365_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"messageId":"MESSAGE_GUID"}'

# === D365: Get execution errors ===
curl -X POST "https://$D365_INSTANCE/data/DataManagementDefinitionGroups/Microsoft.Dynamics.DataEntities.GetExecutionErrors" \
  -H "Authorization: Bearer $D365_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"executionId":"EXECUTION_ID"}'

# === Oracle FBDI: Check ESS job status ===
curl -s "https://$ORACLE_INSTANCE/fscmRestApi/resources/latest/erpintegrations?finder=ESSJobStatusFinder;jobPackageName=oracle/apps/ess/custom" \
  -H "Authorization: Bearer $ORACLE_TOKEN"
```

## Version History & Compatibility

| ERP System | Bulk API Version | Release | Key Change | Impact on Chunking |
|---|---|---|---|---|
| Salesforce | Bulk API 2.0 | v62.0 (2026-02) | PK Chunking default raised to 250K | Larger query chunks possible |
| Salesforce | Bulk API 2.0 | v56.0 (2022-10) | Bulk API 2.0 GA (replaced Bulk API 1.0) | Simpler chunking — no manual batch management |
| NetSuite | SuiteTalk REST | 2024.2 | Idempotency key header support added | Safer retries on bulk REST operations |
| SAP | OData v4 | 2408 | Async OData $batch support | Bulk operations no longer block dialog processes |
| D365 | DMF | PU63 (10.0.38) | Batch Concurrency Control reintroduced | Fine-grained control over parallel tasks |
| D365 | DMF | PU64 (10.0.40) | Auto-retry on batch node restarts | Improved reliability for long-running imports |
| Oracle | FBDI | 26A | File size limit increased to 500 MB for some scenarios | Larger chunks possible for high-volume imports |

[src1, src2, src4, src6]

### Deprecation Policy

Salesforce supports Bulk API 2.0 indefinitely (Bulk API 1.0 is in maintenance mode but not deprecated). SAP OData APIs follow a 2-year deprecation notice policy. D365 DMF evolves with platform updates — breaking changes are documented per PU release. Oracle FBDI evolves with quarterly releases (A/B/C/D suffixes). [src1, src4, src6]

## When to Use / When Not to Use

| Use When | Don't Use When | Use Instead |
|---|---|---|
| Data volume > 2,000 records per operation | Real-time, sub-second latency required | REST API with composite requests |
| Nightly/scheduled data sync between systems | Event-driven integration (react to individual changes) | CDC / Platform Events / Webhooks |
| Master data distribution across multiple ERPs | One-time data migration (initial load) | ETL tools (LTMC, Data Loader, FBDI with staging) |
| High-volume transaction processing (orders, invoices) | Low-volume, high-frequency updates (< 100 records/batch) | Standard REST/OData API |
| Historical data backfill | Bidirectional real-time sync | Dual-write / CDC with conflict resolution |

## Cross-System Comparison

| Capability | Salesforce | NetSuite | SAP S/4HANA | D365 F&O | Oracle ERP Cloud |
|---|---|---|---|---|---|
| **Bulk API maturity** | Excellent (Bulk API 2.0) | Limited (CSV + REST) | Good (OData $batch + LTMC) | Good (DMF + recurring) | Good (FBDI + ESS) |
| **Auto-chunking** | Yes (10K internal batches) | No | No | No | No |
| **Partial success** | Per-record granularity | No (all-or-nothing) | Per-changeset | Per-record (in logs) | No (all-or-nothing) |
| **Concurrency control** | 25 job hard cap | Tier-based (15-55) | System-resource dependent | Configurable (Batch Concurrency Control) | 5 import hard cap |
| **Built-in retry** | 10 retries per batch | None | None | Auto-retry (PU64+) | None |
| **Query chunking** | PK Chunking (250K default) | SuiteQL pagination (1K max) | OData $skip/$top | OData $skip/$top | BICC/BI Publisher |
| **Progress monitoring** | Job status API | Task status API | N/A (check job log) | GetMessageStatus API | ESS job status |
| **Idempotent bulk ops** | Upsert with External ID | External ID (CSV Import) | Not native (middleware) | Alternate key upsert | Not native |
| **Compression support** | No (CSV only) | No | No | ZIP packages | DEFLATE in ZIP |
| **Async processing** | Native | Native (CSV Import) | Async OData (2408+) | Batch mode | ESS jobs |

[src1, src2, src3, src4, src5, src6]

## Important Caveats

- Rate limits and chunk size recommendations are subject to change with each vendor release — always verify against current documentation before implementing [src1, src4, src6]
- Throughput benchmarks (records/second) vary dramatically based on record complexity, custom logic (triggers, workflows, validations), and time of day — the figures in this card are baseline estimates for simple records without custom automation
- Edition-specific differences exist for all platforms — Salesforce Developer edition has drastically lower limits (15K API calls/24h); NetSuite account tiers determine concurrent threads; D365 limits scale with license count
- Sandbox environments have different performance characteristics than production — always load-test with production-representative data volumes [src7]
- These patterns assume server-to-server integration — browser-based or mobile client bulk operations have additional constraints (CORS, payload size, session timeout)
- Oracle FBDI limits may vary by functional module — procurement, financials, and supply chain may have different per-import record caps [src5, src6]

## Related Units

- [ERP Bulk Import Comparison](/business/erp-integration/erp-bulk-import-comparison/2026)
- [ERP Rate Limits Comparison](/business/erp-integration/erp-rate-limits-comparison/2026)
- [Batch vs Real-time Integration](/business/erp-integration/batch-vs-realtime-integration/2026)
- [Error Handling & Dead Letter Queues](/business/erp-integration/error-handling-dead-letter-queues/2026)
- [Idempotency in ERP Integrations](/business/erp-integration/idempotency-erp-integrations/2026)
- [Salesforce Bulk API Capabilities](/business/erp-integration/salesforce-bulk-api-capabilities/2026)
- [D365 DMF Import/Export](/business/erp-integration/dynamics-365-dmf-import-export/2026)
- [Change Data Capture for ERP](/business/erp-integration/change-data-capture-erp/2026)