Infor Data Lake: Bulk Data Extraction via Compass Queries

TL;DR

• Bottom line: Use the Compass v2 API (Data Fabric Suite) to submit asynchronous SQL SELECT queries against Infor Data Lake, then paginate through results using offset/limit. The old ION API Suite Compass endpoints are deprecated. [src1]
• Key limit: 100,000 rows or 10 MB per result page; 60-minute query timeout; 100 job submissions per tenant per minute. [src1, src2]
• Watch out for: Compass only supports SELECT queries on flat data (newline-delimited JSON and DSV). Hierarchical or nested documents require the separate Data Lake API. [src4, src5]
• Best for: Scheduled bulk extraction, ETL pipelines, ad-hoc analytics, and BI tool integration (via JDBC driver) against data already published to Infor Data Lake. [src3]
• Authentication: OAuth 2.0 via ION API credentials (.ionapi file). Service account access key (saak) and secret key (sask) exchange for bearer tokens with ~2-hour lifetime. [src1, src6]

System Profile

Infor Data Lake is the centralized data repository within Infor OS (the cloud platform underpinning all Infor CloudSuites). Data from Infor applications — M3, LN, CloudSuite Industrial, CloudSuite Distribution, and others — is automatically published to Data Lake via ION messaging. The Compass query engine sits atop Data Lake and provides SQL-like query capabilities for extracting this data. [src3]

This card covers the Compass v2 REST API available through the Infor Data Fabric Suite, which replaced the deprecated ION API Suite Compass endpoints (removed April 2025). It does NOT cover the Data Lake API (for hierarchical document retrieval), ION Data Lake Flows (for automated S3-based data routing), or the Compass UI within the Data Fabric application. [src1, src5]

API Surfaces & Capabilities

Infor provides multiple ways to extract data from Data Lake. The Compass API is the primary programmatic interface for SQL-based querying. [src5]

Rate Limits & Quotas

Per-Request Limits

Rolling / Daily Limits

Authentication

All Compass API calls go through the ION API Gateway, which uses OAuth 2.0 for authentication. Credentials are provisioned as a .ionapi file downloaded from the Infor OS Portal. [src1, src6]

Authentication Gotchas

• The .ionapi file contains ALL credentials (tenant ID, client ID/secret, service account keys, OAuth endpoints). Treat it like a private key — never commit to source control. [src6, src8]
• Token endpoint is constructed from .ionapi fields: combine pu (base URL) + ot (token path). The saak goes as username and sask as password in the token request. [src6]
• Tokens expire after ~2 hours. Always implement token caching and refresh logic. [src6]
• Compass v2 is in the Data Fabric Suite, not the ION API Suite. Ensure the authorized app has access to the Data Fabric API suite. [src1]

Constraints

• SELECT only — Compass does not support CREATE, INSERT, UPDATE, DELETE, or DROP statements. All write operations must use ION messaging or other Infor APIs. [src4]
• Flat data only — Compass API can only query newline-delimited JSON and DSV objects. Hierarchical documents require the Data Lake API. [src5]
• Results are static snapshots — Once a query completes, the result set is frozen. New data arriving after submission is NOT included. [src2]
• 60-minute timeout with compute charge — Timed-out queries are billed for full compute time despite returning no results. [src3, src7]
• No semi-colon multi-query support — Each query requires a separate job submission. [src4]
• Tenant-scoped isolation — Query results are only accessible to the submitting tenant and principal ID. [src1]
• AWS GovCloud not supported — Compass API is not available in AWS GovCloud regions. [src1]
• Results expire after ~20 hours — Must retrieve results before expiry. [src2]

Integration Pattern Decision Tree

START — Extract data from Infor Data Lake
|-- What data format?
|   |-- Flat (DSV or newline-delimited JSON)
|   |   |-- Data volume < 100K rows per query?
|   |   |   |-- YES --> Compass v2 API (single page result)
|   |   |   +-- NO --> Compass v2 API with pagination (offset/limit)
|   |   |-- Need SQL joins across tables?
|   |   |   |-- YES --> Compass v2 API (supports JOINs) [src4]
|   |   |   +-- NO --> Data Lake API may also work for simple lookups [src5]
|   |   +-- Need BI tool integration?
|   |       |-- YES --> Compass JDBC driver (DBeaver, Tableau, Power BI) [src3]
|   |       +-- NO --> Compass v2 REST API
|   +-- Hierarchical/nested JSON
|       +-- Data Lake API (not Compass) [src5]
|-- What's the extraction pattern?
|   |-- One-time or ad-hoc extraction
|   |   +-- Compass v2 API or Compass UI [src3]
|   |-- Scheduled recurring ETL
|   |   |-- < 60 min query time?
|   |   |   |-- YES --> Compass v2 API with job scheduler
|   |   |   +-- NO --> ION Data Lake Flows (S3 export) [src5]
|   |   +-- Need incremental/delta loads?
|   |       |-- YES --> Filter on infor.lastModified() property [src5]
|   |       +-- NO --> Full table query
|   +-- Automated S3 export with CDC
|       +-- ION Data Lake Flows (first load = full, subsequent = delta) [src5]
|-- Authentication approach?
|   |-- Service account (unattended) --> OAuth 2.0 with saak/sask from .ionapi [src6]
|   +-- User context (interactive) --> OAuth 2.0 authorization code flow
+-- Error tolerance?
    |-- Zero-loss --> Implement idempotency via infor.DataObjectId tracking
    +-- Best-effort --> Simple retry on FAILED status

Quick Reference

Step-by-Step Integration Guide

1. Obtain ION API credentials

Navigate to Infor OS Portal > ION API > Authorized Apps. Create or select a Backend Service authorized app. Click "Download Credentials" and select "Create Service Account." Save the .ionapi file securely. [src6, src8]

{
  "ti": "TENANT_ID",
  "cn": "common_name",
  "ci": "client_id",
  "cs": "client_secret",
  "iu": "https://mingle-ionapi.inforcloudsuite.com/TENANT_ID",
  "pu": "https://mingle-sso.inforcloudsuite.com:443/TENANT_ID/as/",
  "oa": "authorization.oauth2",
  "ot": "token.oauth2",
  "or": "revoke_token.oauth2",
  "saak": "SERVICE_ACCOUNT_ACCESS_KEY",
  "sask": "SERVICE_ACCOUNT_SECRET_KEY"
}

Verify: Open the .ionapi file and confirm all fields are populated. The saak and sask fields must be present.

2. Acquire OAuth 2.0 bearer token

Use the token endpoint URL (constructed from pu + ot) with the service account credentials. [src6]

TOKEN_URL="https://mingle-sso.inforcloudsuite.com:443/TENANT_ID/as/token.oauth2"

ACCESS_TOKEN=$(curl -s -X POST "$TOKEN_URL" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=password" \
  -d "username=SERVICE_ACCOUNT_ACCESS_KEY" \
  -d "password=SERVICE_ACCOUNT_SECRET_KEY" \
  -d "client_id=CLIENT_ID" \
  -d "client_secret=CLIENT_SECRET" \
  | jq -r '.access_token')

Verify: echo $ACCESS_TOKEN should output a JWT string. Token should not be null or empty.

3. Submit a Compass query

POST your SQL SELECT query to the /v2/compass/jobs endpoint with Content-Type text/plain. [src1, src2]

API_BASE="https://mingle-ionapi.inforcloudsuite.com/TENANT_ID/IONSERVICES/datafabric"

QUERY_ID=$(curl -s -X POST "$API_BASE/v2/compass/jobs" \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: text/plain" \
  -d "SELECT CONO, ITNO, ITDS, FUDS FROM MITMAS ORDER BY CONO, ITNO" \
  | jq -r '.queryId')

Verify: Response contains a queryId string (UUID format).

4. Poll for query completion

Check the status endpoint until the query reaches FINISHED, FAILED, or CANCELLED state. [src2]

while true; do
  STATUS=$(curl -s -X POST "$API_BASE/v2/compass/$QUERY_ID/status" \
    -H "Authorization: Bearer $ACCESS_TOKEN" | jq -r '.status')
  echo "Status: $STATUS"
  [ "$STATUS" = "FINISHED" ] || [ "$STATUS" = "FAILED" ] && break
  sleep 5
done

Verify: Status transitions from RUNNING to FINISHED. The rowCount field shows total available rows.

5. Retrieve paginated results

Use offset and limit parameters to page through the result set. Maximum 100,000 rows or 10 MB per page. [src2]

curl -s "$API_BASE/v2/compass/jobs/$QUERY_ID/result?offset=0&limit=10000" \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Accept: text/csv" \
  -H "Accept-Encoding: gzip" \
  -o "result_page_0.csv"

Verify: Output file contains header row + data rows matching expected count.

6. Implement full pagination loop

For result sets larger than one page, iterate with increasing offsets until all rows are retrieved. [src2]

PAGE_SIZE=50000; OFFSET=0; PAGE=0
while [ $OFFSET -lt $TOTAL_ROWS ]; do
  curl -s "$API_BASE/v2/compass/jobs/$QUERY_ID/result?offset=$OFFSET&limit=$PAGE_SIZE" \
    -H "Authorization: Bearer $ACCESS_TOKEN" -H "Accept: text/csv" \
    -o "result_page_${PAGE}.csv"
  OFFSET=$((OFFSET + PAGE_SIZE)); PAGE=$((PAGE + 1))
done

Verify: Total rows across all pages equals the rowCount from status response.

Code Examples

Python: Bulk extract with pagination

# Input:  .ionapi credentials file path, SQL query
# Output: All query results as a pandas DataFrame

import json, time, requests, pandas as pd
from io import StringIO

def compass_query(ionapi_path, sql, page_size=50000):
    creds = json.load(open(ionapi_path))
    # Get token
    token_resp = requests.post(f"{creds['pu']}{creds['ot']}", data={
        'grant_type': 'password', 'username': creds['saak'],
        'password': creds['sask'], 'client_id': creds['ci'],
        'client_secret': creds['cs']})
    token = token_resp.json()['access_token']
    base = f"{creds['iu']}/IONSERVICES/datafabric"
    hdrs = {'Authorization': f'Bearer {token}'}
    # Submit query
    qid = requests.post(f"{base}/v2/compass/jobs",
        headers={**hdrs, 'Content-Type': 'text/plain'}, data=sql).json()['queryId']
    # Poll status
    while True:
        s = requests.post(f"{base}/v2/compass/{qid}/status", headers=hdrs).json()
        if s['status'] == 'FINISHED': break
        if s['status'] in ('FAILED','CANCELLED'): raise RuntimeError(s)
        time.sleep(3)
    # Paginate results
    dfs, offset = [], 0
    while offset < s.get('rowCount', 0):
        r = requests.get(f"{base}/v2/compass/jobs/{qid}/result",
            headers={**hdrs, 'Accept': 'text/csv'},
            params={'offset': offset, 'limit': page_size})
        dfs.append(pd.read_csv(StringIO(r.text))); offset += page_size
    return pd.concat(dfs, ignore_index=True) if dfs else pd.DataFrame()

JavaScript/Node.js: Compass query with async/await

// Input:  .ionapi file path, SQL query string
// Output: Array of result objects from Data Lake
// npm install axios@1
const axios = require('axios');
const fs = require('fs');

async function compassQuery(ionapiPath, sql, pageSize = 50000) {
  const creds = JSON.parse(fs.readFileSync(ionapiPath, 'utf-8'));
  const tokenResp = await axios.post(`${creds.pu}${creds.ot}`,
    new URLSearchParams({ grant_type: 'password', username: creds.saak,
      password: creds.sask, client_id: creds.ci, client_secret: creds.cs }));
  const token = tokenResp.data.access_token;
  const base = `${creds.iu}/IONSERVICES/datafabric`;
  const hdrs = { Authorization: `Bearer ${token}` };
  // Submit
  const { data: { queryId } } = await axios.post(`${base}/v2/compass/jobs`,
    sql, { headers: { ...hdrs, 'Content-Type': 'text/plain' } });
  // Poll
  let status = 'RUNNING', rowCount = 0;
  while (status === 'RUNNING') {
    await new Promise(r => setTimeout(r, 3000));
    const s = await axios.post(`${base}/v2/compass/${queryId}/status`, null, { headers: hdrs });
    status = s.data.status; rowCount = s.data.rowCount || 0;
  }
  if (status !== 'FINISHED') throw new Error(`Query ${status}`);
  // Paginate (NDJSON)
  const rows = []; let offset = 0;
  while (offset < rowCount) {
    const r = await axios.get(`${base}/v2/compass/jobs/${queryId}/result`,
      { headers: { ...hdrs, Accept: 'application/x-ndjson' },
        params: { offset, limit: pageSize } });
    rows.push(...r.data.trim().split('\n').map(l => JSON.parse(l)));
    offset += pageSize;
  }
  return rows;
}

cURL: Quick Compass API test

# Input:  .ionapi credentials, SQL query
# Output: First page of query results

# Step 1: Get token
TOKEN=$(curl -s -X POST \
  "https://mingle-sso.inforcloudsuite.com:443/TENANT/as/token.oauth2" \
  -d "grant_type=password&username=SAAK&password=SASK&client_id=CI&client_secret=CS" \
  | jq -r '.access_token')

# Step 2: Submit query
BASE="https://mingle-ionapi.inforcloudsuite.com/TENANT/IONSERVICES/datafabric"
QID=$(curl -s -X POST "$BASE/v2/compass/jobs" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: text/plain" \
  -d "SELECT TOP 10 CONO, ITNO, ITDS FROM MITMAS" | jq -r '.queryId')

# Step 3: Wait and check status
sleep 10
curl -s -X POST "$BASE/v2/compass/$QID/status" \
  -H "Authorization: Bearer $TOKEN" | jq .

# Step 4: Get results as CSV
curl -s "$BASE/v2/compass/jobs/$QID/result?offset=0&limit=100" \
  -H "Authorization: Bearer $TOKEN" -H "Accept: text/csv"

Data Mapping

Infor Data Lake Property Reference

Data Type Gotchas

• Field names are ERP-specific: Infor M3 uses short cryptic names (ITNO = Item Number, CONO = Company). CloudSuite Industrial uses different conventions. No standard cross-product naming. [src6]
• Timestamps in Data Lake may differ from source: infor.lastModified() reflects when data arrived in Data Lake, NOT when the source record was last modified. [src5]
• DSV format varies by data source: Different Infor products publish in different formats. The Compass query engine normalizes this. [src5]
• NULL handling: Empty values in DSV data may appear as empty strings rather than SQL NULL. Use COALESCE or explicit null checks. [src4]

Error Handling & Failure Points

Common Error Codes

Failure Points in Production

• Token expiry during long pagination: A query with millions of rows may take longer to paginate than the ~2-hour token lifetime. Fix: Check token expiry before each page request. Implement proactive token refresh when remaining lifetime < 10 minutes. [src6]
• 60-minute timeout on complex queries: Large table scans without WHERE clauses easily exceed the timeout, and compute time is still billed. Fix: Include WHERE clauses. Use infor.lastModified() for incremental extraction. Test in Compass UI first. [src3, src7]
• Results expire after ~20 hours: If you don't retrieve results within 20 hours, data is purged. Fix: Start retrieval immediately after FINISHED status. [src2]
• Deprecated ION API Suite endpoints: Code using old endpoints (removed April 2025) will fail. Fix: Migrate from /IONSERVICES/datalakeapi to /IONSERVICES/datafabric. [src1]
• .ionapi credential rotation: When ION API credentials are regenerated, all integrations using old credentials break silently. Fix: Monitor for 401 errors. Store credentials in a secrets manager. [src6]

Anti-Patterns

Wrong: Polling status in a tight loop

# BAD -- burns through 1,000/min status rate limit in seconds
while True:
    status = check_status(query_id)
    if status == 'FINISHED':
        break

Correct: Poll with exponential backoff

# GOOD -- respects rate limits, starts fast and backs off
import time
delay = 2
max_delay = 30
while True:
    status = check_status(query_id)
    if status in ('FINISHED', 'FAILED', 'CANCELLED'):
        break
    time.sleep(delay)
    delay = min(delay * 1.5, max_delay)

Wrong: Requesting a new OAuth token for every API call

# BAD -- wastes time, may trigger auth server rate limits
for page in range(total_pages):
    token = get_access_token(creds)  # new token per page!
    get_results(token, query_id, offset=page * page_size)

Correct: Cache token and reuse until near expiry

# GOOD -- token cached, only refreshed when near expiry
token = get_access_token(creds)
token_expiry = time.time() + 7000  # ~2h minus buffer
for page in range(total_pages):
    if time.time() > token_expiry:
        token = get_access_token(creds)
        token_expiry = time.time() + 7000
    get_results(token, query_id, offset=page * page_size)

Wrong: Full table scan for incremental loads

-- BAD -- scans entire table every time, may timeout
SELECT * FROM MITMAS ORDER BY ITNO

Correct: Use infor.lastModified() for incremental extraction

-- GOOD -- only extracts records added/modified since last run
SELECT CONO, ITNO, ITDS, FUDS FROM MITMAS
WHERE dl_document_indexed_date > '2026-02-28T00:00:00Z'
ORDER BY ITNO

Common Pitfalls

• Using the deprecated ION API Suite Compass endpoint: The old endpoint at /IONSERVICES/datalakeapi was removed April 2025. Fix: Migrate to Data Fabric Suite endpoint at /IONSERVICES/datafabric. [src1]
• Forgetting Content-Type: text/plain on job submission: The v2 /jobs endpoint requires text/plain. Sending application/json fails. Fix: Always set Content-Type: text/plain when POSTing to /v2/compass/jobs. [src1]
• Assuming real-time data: Data Lake receives data via ION messaging with latency. Fix: For real-time needs, use ION API direct calls to the source ERP. [src5]
• Not handling FAILED status from /result: Error details are in the /result response, not /status. Fix: Always call /result for FAILED queries to get the error message. [src2]
• Ignoring compute time metering: Every query consumes metered compute time, including timed-out queries. Fix: Test in Compass UI first. Add WHERE clauses. Use LIMIT during dev. [src7]
• Page size too large for wide tables: 100K rows on 50+ column tables may exceed 10 MB limit. Fix: Select only needed columns. Start with smaller page size (10K) and adjust. [src2]

Diagnostic Commands

# Test OAuth token acquisition
curl -s -X POST "https://mingle-sso.inforcloudsuite.com:443/TENANT/as/token.oauth2" \
  -d "grant_type=password&username=SAAK&password=SASK&client_id=CI&client_secret=CS" \
  | jq '{access_token: .access_token[:20], expires_in: .expires_in, token_type: .token_type}'

# Submit a minimal test query
curl -s -X POST "$BASE/v2/compass/jobs" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: text/plain" \
  -d "SELECT TOP 1 * FROM MITMAS" | jq .

# Check query status
curl -s -X POST "$BASE/v2/compass/$QID/status" \
  -H "Authorization: Bearer $TOKEN" | jq .

# Cancel a running query
curl -s -X PUT "$BASE/v2/compass/$QID/cancel" \
  -H "Authorization: Bearer $TOKEN" | jq .

Version History & Compatibility

Deprecation Policy

Infor deprecates API endpoints with advance notice through release notes and Documentation Central. The Compass v1/v2 ION API Suite endpoints were deprecated with a migration window of approximately 12-18 months before removal. Always monitor Infor's release notes for upcoming changes. [src1]

When to Use / When Not to Use

Important Caveats

• Compass API is limited to SELECT queries on flat data (DSV and newline-delimited JSON). It does NOT support DML statements or hierarchical document querying. [src4, src5]
• The 60-minute query timeout is a hard limit. Timed-out queries still consume compute time. Always test query performance in the Compass UI before automating via API. [src3, src7]
• Data in the Data Lake may not be real-time. Latency depends on ION publishing configuration and can range from minutes to hours. [src5]
• Rate limits are per-tenant, not per-user. Heavy usage by one integration can impact all others in the same tenant. [src1, src2]
• Compute time and egress are metered and may incur additional costs depending on your Infor OS subscription tier. [src7]