Idempotency Patterns for APIs and Distributed Systems

TL;DR

• Bottom line: Combine an idempotency key header with an atomic check-and-execute pattern (DB unique constraint or Redis SETNX) to make any mutating API endpoint safely retryable.
• Key tool/command: Idempotency-Key HTTP header + database/Redis store with TTL
• Watch out for: Race conditions between the idempotency check and the mutation -- these must be atomic (single transaction or distributed lock).
• Works with: Any HTTP API framework, any database (PostgreSQL, MySQL, DynamoDB), Redis, message brokers (Kafka, SQS, RabbitMQ).

Constraints

• Idempotency check MUST happen BEFORE mutation -- checking after write is useless
• Idempotency key + execute MUST be atomic (single transaction or distributed lock) to prevent race conditions
• Idempotency keys MUST have a TTL (24h-30d typical) -- unbounded storage causes OOM or table bloat
• NEVER reuse an idempotency key with different request parameters -- servers must reject with 422
• Client-generated keys MUST use cryptographically random UUIDs (v4) -- sequential IDs enable guessing attacks

Quick Reference

Decision Tree

START
|-- Is the operation naturally idempotent (GET, PUT with full resource, DELETE)?
|   |-- YES --> No idempotency mechanism needed; ensure PUT replaces full resource
|   +-- NO (POST, PATCH, or side-effect-producing operation) |
|       |-- Is this an API endpoint (HTTP)?
|       |   |-- YES --> Use Idempotency-Key header pattern
|       |   |   |-- Is latency critical (<10ms overhead)?
|       |   |   |   |-- YES --> Use Redis SETNX for idempotency store
|       |   |   |   +-- NO --> Use database table with unique constraint
|       |   |   +-- Do you need to store the full response?
|       |   |       |-- YES --> Store response_code + response_body (Stripe pattern)
|       |   |       +-- NO --> Store only key + status (lighter)
|       |   +-- NO (message queue / event consumer)?
|       |       |-- Does the broker support native dedup (SQS FIFO, Kafka EOS)?
|       |       |   |-- YES --> Enable broker-level dedup + application-level as defense-in-depth
|       |       |   +-- NO --> Implement idempotent consumer (inbox pattern with DB)
|       |       +-- Are messages processed in transactions?
|       |           |-- YES --> Store message ID in same transaction as business logic
|       |           +-- NO --> Use separate dedup table with TTL cleanup
+-- DEFAULT --> Add Idempotency-Key header middleware + PostgreSQL store

Step-by-Step Guide

1. Create the idempotency key store

Create a database table to track idempotency keys, their status, and cached responses. [src2]

CREATE TABLE idempotency_keys (
    id              BIGSERIAL PRIMARY KEY,
    idempotency_key TEXT NOT NULL,
    user_id         TEXT NOT NULL,
    request_method  TEXT NOT NULL,
    request_path    TEXT NOT NULL,
    request_hash    TEXT NOT NULL,
    status          TEXT NOT NULL DEFAULT 'started',
    response_code   INT,
    response_body   JSONB,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    locked_at       TIMESTAMPTZ,
    CONSTRAINT uq_idempotency UNIQUE (user_id, idempotency_key)
);
CREATE INDEX idx_idempotency_created ON idempotency_keys (created_at);

Verify: SELECT COUNT(*) FROM idempotency_keys; --> expected: 0

2. Implement idempotency middleware

Add middleware that intercepts incoming requests, checks for the Idempotency-Key header, and either returns a cached response or proceeds with processing. [src1] [src4]

async function idempotencyMiddleware(req, res, next) {
  const key = req.headers['idempotency-key'];
  if (!key || req.method === 'GET') return next();
  const hash = crypto.createHash('sha256')
    .update(JSON.stringify(req.body)).digest('hex');
  const userId = req.user?.id || req.ip;
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    const { rows } = await client.query(`
      INSERT INTO idempotency_keys (idempotency_key, user_id, request_method, request_path, request_hash, status)
      VALUES ($1, $2, $3, $4, $5, 'processing')
      ON CONFLICT (user_id, idempotency_key) DO UPDATE SET locked_at = now()
      RETURNING status, response_code, response_body, request_hash
    `, [key, userId, req.method, req.path, hash]);
    const existing = rows[0];
    if (existing.status === 'finished') {
      if (existing.request_hash !== hash) {
        await client.query('ROLLBACK');
        return res.status(422).json({ error: 'Idempotency key reused with different parameters' });
      }
      await client.query('COMMIT');
      return res.status(existing.response_code).json(existing.response_body);
    }
    await client.query('COMMIT');
    next();
  } catch (err) { await client.query('ROLLBACK'); next(err); }
  finally { client.release(); }
}

Verify: Send the same POST twice with the same Idempotency-Key header --> second response is identical with no side effects.

3. Add TTL cleanup

Schedule periodic cleanup of expired idempotency keys to prevent table bloat. [src3]

DELETE FROM idempotency_keys WHERE created_at < now() - INTERVAL '24 hours';

Verify: SELECT COUNT(*) FROM idempotency_keys WHERE created_at < now() - INTERVAL '24 hours'; --> expected: 0

4. Implement message-level deduplication

For event-driven systems, add an inbox table and check message IDs before processing. [src5]

CREATE TABLE processed_messages (
    subscriber_id TEXT NOT NULL,
    message_id    TEXT NOT NULL,
    processed_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (subscriber_id, message_id)
);

Verify: Publish the same message twice --> business logic executes exactly once.

Code Examples

Node.js/Express: Redis-Backed Idempotency Middleware

// Input:  HTTP request with Idempotency-Key header
// Output: Cached response on retry, fresh response on first call

const crypto = require('crypto');
const Redis = require('ioredis');        // ioredis ^5.0.0
const redis = new Redis(process.env.REDIS_URL);
const IDEM_TTL = 86400; // 24 hours

async function redisIdempotency(req, res, next) {
  const key = req.headers['idempotency-key'];
  if (!key || req.method === 'GET') return next();
  const hash = crypto.createHash('sha256')
    .update(JSON.stringify(req.body)).digest('hex');
  const redisKey = `idem:${req.user?.id || req.ip}:${key}`;
  const locked = await redis.set(redisKey, JSON.stringify({
    status: 'processing', hash
  }), 'EX', IDEM_TTL, 'NX');
  if (!locked) {
    const existing = JSON.parse(await redis.get(redisKey));
    if (existing?.hash !== hash)
      return res.status(422).json({ error: 'Key reused with different payload' });
    if (existing?.status === 'finished')
      return res.status(existing.code).json(existing.body);
    return res.status(409).json({ error: 'Request is still processing' });
  }
  const originalJson = res.json.bind(res);
  res.json = async (body) => {
    await redis.set(redisKey, JSON.stringify({
      status: 'finished', hash, code: res.statusCode, body
    }), 'EX', IDEM_TTL);
    return originalJson(body);
  };
  next();
}

Python/FastAPI: Database-Backed Idempotency Decorator

# Input:  FastAPI request with Idempotency-Key header
# Output: Cached response on retry, fresh response on first call

import hashlib, json
from functools import wraps
from fastapi import Request, HTTPException
from sqlalchemy import text        # SQLAlchemy ^2.0

def idempotent(func):
    @wraps(func)
    async def wrapper(request: Request, *args, **kwargs):
        key = request.headers.get("Idempotency-Key")
        if not key:
            return await func(request, *args, **kwargs)
        body = await request.body()
        req_hash = hashlib.sha256(body).hexdigest()
        user_id = getattr(request.state, "user_id", request.client.host)
        async with db.begin() as conn:
            result = await conn.execute(text("""
                INSERT INTO idempotency_keys
                    (idempotency_key, user_id, request_method,
                     request_path, request_hash, status)
                VALUES (:key, :uid, :method, :path, :hash, 'processing')
                ON CONFLICT (user_id, idempotency_key)
                DO UPDATE SET locked_at = now()
                RETURNING status, response_code, response_body, request_hash
            """), {"key": key, "uid": user_id, "method": request.method,
                   "path": request.url.path, "hash": req_hash})
            row = result.fetchone()
        if row.status == "finished":
            if row.request_hash != req_hash:
                raise HTTPException(422, "Key reused with different params")
            return json.loads(row.response_body)
        response = await func(request, *args, **kwargs)
        async with db.begin() as conn:
            await conn.execute(text("""
                UPDATE idempotency_keys SET status='finished',
                  response_code=:code, response_body=:body
                WHERE user_id=:uid AND idempotency_key=:key
            """), {"code": 200, "body": json.dumps(response),
                   "uid": user_id, "key": key})
        return response
    return wrapper

Go: Idempotency Middleware with PostgreSQL

// Input:  HTTP request with Idempotency-Key header
// Output: Cached response on replay, fresh response on first call

func IdempotencyMiddleware(db *sql.DB) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            key := r.Header.Get("Idempotency-Key")
            if key == "" || r.Method == "GET" {
                next.ServeHTTP(w, r); return
            }
            body, _ := io.ReadAll(r.Body)
            hash := sha256.Sum256(body)
            hashHex := hex.EncodeToString(hash[:])
            tx, _ := db.Begin()
            var status, reqHash string
            var respCode int; var respBody []byte
            err := tx.QueryRow(`
                INSERT INTO idempotency_keys
                  (idempotency_key, user_id, request_method,
                   request_path, request_hash, status)
                VALUES ($1, $2, $3, $4, $5, 'processing')
                ON CONFLICT (user_id, idempotency_key)
                DO UPDATE SET locked_at = now()
                RETURNING status, response_code, response_body, request_hash`,
                key, r.RemoteAddr, r.Method, r.URL.Path, hashHex,
            ).Scan(&status, &respCode, &respBody, &reqHash)
            if err != nil { tx.Rollback(); http.Error(w, "Error", 500); return }
            tx.Commit()
            if status == "finished" {
                if reqHash != hashHex { http.Error(w, "Key reused", 422); return }
                w.Header().Set("Content-Type", "application/json")
                w.WriteHeader(respCode); w.Write(respBody); return
            }
            next.ServeHTTP(w, r)
        })
    }
}

Anti-Patterns

Wrong: Checking idempotency AFTER mutation

// BAD -- mutation happens before dedup check; damage already done
app.post('/api/orders', async (req, res) => {
  const order = await db.query('INSERT INTO orders ... RETURNING *');  // side effect!
  const existing = await db.query(
    'SELECT * FROM idempotency_keys WHERE key = $1', [req.headers['idempotency-key']]
  );
  if (existing.rows.length) return res.json(existing.rows[0].response);
});

Correct: Atomic check-then-execute

// GOOD -- idempotency check is first, within the same transaction
app.post('/api/orders', async (req, res) => {
  const client = await pool.connect();
  await client.query('BEGIN');
  const { rows } = await client.query(
    `INSERT INTO idempotency_keys (...) VALUES (...)
     ON CONFLICT (...) DO UPDATE SET locked_at = now()
     RETURNING status, response_code, response_body`, [...]
  );
  if (rows[0].status === 'finished') {
    await client.query('COMMIT');
    return res.status(rows[0].response_code).json(rows[0].response_body);
  }
  const order = await client.query('INSERT INTO orders ... RETURNING *');
  await client.query('UPDATE idempotency_keys SET status=... WHERE ...');
  await client.query('COMMIT');
  res.status(201).json(order.rows[0]);
});

Wrong: No TTL on idempotency keys

// BAD -- keys accumulate forever, table grows unbounded
await db.query(
  `INSERT INTO idempotency_keys (key, response) VALUES ($1, $2)`,
  [key, response]
);
// No cleanup job, no TTL, no partition pruning

Correct: TTL with automated cleanup

// GOOD -- keys expire and are cleaned up
// Option A: Application-level cleanup (cron)
cron.schedule('0 * * * *', () =>
  db.query(`DELETE FROM idempotency_keys WHERE created_at < now() - INTERVAL '24h'`)
);
// Option B: Redis with built-in TTL
await redis.set(`idem:${key}`, response, 'EX', 86400);
// Option C: DynamoDB with TTL attribute
await dynamodb.put({ Item: { pk: key, ttl: Math.floor(Date.now()/1000) + 86400 } });

Wrong: Non-atomic check-and-execute (race condition)

// BAD -- race condition: two concurrent requests both pass the check
const existing = await db.query('SELECT * FROM idempotency_keys WHERE key=$1', [key]);
if (existing.rows.length > 0) return res.json(existing.rows[0].response);
// WINDOW: another request can arrive HERE before insert
await db.query('INSERT INTO idempotency_keys (key) VALUES ($1)', [key]);
await processPayment(); // Both requests process the payment!

Correct: Atomic upsert eliminates race window

// GOOD -- INSERT ON CONFLICT is atomic; no race window
const { rows } = await db.query(`
  INSERT INTO idempotency_keys (idempotency_key, user_id, status)
  VALUES ($1, $2, 'processing')
  ON CONFLICT (user_id, idempotency_key) DO UPDATE SET locked_at = now()
  RETURNING status, response_code, response_body
`, [key, userId]);

Common Pitfalls

• Client-generated UUIDs without server validation: Clients can send malformed or predictable keys, enabling replay attacks. Fix: validate key format server-side and enforce max length (255 chars). [src3]
• Storing keys without user/client scoping: One user could replay another user's key. Fix: always use (user_id, idempotency_key) as the composite unique constraint. [src2]
• Returning 200 for in-progress requests: If a retry arrives while the original is still processing, returning success is wrong. Fix: return 409 Conflict with Retry-After header. [src4]
• Not hashing request body for parameter validation: Allows the same key with different payloads. Fix: store SHA-256 hash of request body and compare on retry; reject with 422 if mismatched. [src1]
• Idempotency at the wrong layer: Adding API idempotency but calling non-idempotent downstream services. Fix: propagate idempotency keys to downstream APIs. [src2]
• Message dedup relying solely on broker features: SQS standard queues and Kafka without EOS can still deliver duplicates. Fix: always implement application-level dedup (inbox pattern) as defense in depth. [src5]

Diagnostic Commands

# Check idempotency key table size and oldest key
psql -c "SELECT COUNT(*), MIN(created_at), MAX(created_at) FROM idempotency_keys;"

# Find stuck/orphaned processing keys (older than 5 minutes)
psql -c "SELECT * FROM idempotency_keys WHERE status = 'processing' AND locked_at < now() - INTERVAL '5 minutes';"

# Check Redis idempotency key count and memory
redis-cli KEYS "idem:*" | wc -l
redis-cli INFO memory | grep used_memory_human

# Verify TTL is set on a specific Redis key
redis-cli TTL "idem:user123:8e03978e-40d5-43e8-bc93-6894a57f9324"

# Check for duplicate processing in message consumer logs
grep -c "Duplicate message" /var/log/consumer.log

When to Use / When Not to Use

Important Caveats

• The IETF Idempotency-Key header spec (draft-07) is not yet an RFC -- it is a stable draft but not finalized; follow updates at the IETF HTTPAPI WG
• Redis-based idempotency loses state on restart if persistence is disabled -- use AOF or RDB persistence, or prefer database-backed storage for critical operations (payments)
• Idempotency does not equal exactly-once processing -- it provides at-most-once side effects for a given key, but the response may be lost if the connection drops after server-side execution
• Different cloud providers have different built-in idempotency: AWS Lambda Powertools uses DynamoDB, Azure Functions use Durable Entities, GCP Cloud Functions require manual implementation
• Composite keys (user_id + idempotency_key) are essential for multi-tenant systems -- without user scoping, one tenant can interfere with another's operations