--- # === IDENTITY === id: software/patterns/webhook-implementation/2026 canonical_question: "How do I implement a reliable webhook system?" aliases: - "How to build webhooks with HMAC signature verification" - "Webhook sender and receiver implementation guide" - "How to implement idempotent webhook processing" - "Webhook retry with exponential backoff" - "How to secure webhooks with HMAC-SHA256" - "Webhook best practices for production" entity_type: software_reference domain: software > patterns > webhook implementation region: global jurisdiction: global temporal_scope: 2020-2026 # === VERIFICATION === last_verified: 2026-02-24 confidence: 0.90 version: 1.0 first_published: 2026-02-24 # === TEMPORAL VALIDITY === temporal_validity: status: stable last_breaking_change: "Standard Webhooks spec v1.0, 2023; pattern is protocol-level and language-agnostic" next_review: 2026-08-23 change_sensitivity: low # === CONSTRAINTS === constraints: - "Always sign payloads with HMAC-SHA256 using a per-endpoint secret -- unsigned webhooks are trivially spoofable" - "Always use constant-time comparison (crypto.timingSafeEqual / hmac.compare_digest / hmac.Equal) -- timing attacks leak secret bytes" - "Receivers must return 2xx within 5-10 seconds -- process asynchronously via queue, never synchronously" - "Exactly-once delivery is impossible over HTTP -- design for at-least-once with idempotency keys" - "Verify signatures against the raw request body bytes -- re-serialized JSON changes byte order and breaks HMAC" - "Include timestamps in signatures and reject payloads older than 5 minutes to prevent replay attacks" # === SKIP CONDITIONS === skip_this_unit_if: - condition: "Need to consume webhooks from a specific provider (Stripe, GitHub, Shopify)" use_instead: "That provider's webhook integration docs -- each has unique header names and signing schemes" - condition: "Need real-time bidirectional communication between client and server" use_instead: "WebSockets or Server-Sent Events (SSE)" - condition: "Need guaranteed strict ordering of all events across all consumers" use_instead: "Event streaming with Kafka, Pulsar, or similar message broker" # === AGENT HINTS === inputs_needed: - key: role question: "Are you building the webhook sender (provider), receiver (consumer), or both?" type: choice options: ["sender", "receiver", "both"] - key: delivery_guarantee question: "What delivery guarantee do you need?" type: choice options: ["at-least-once", "at-most-once"] - key: scale question: "What is your expected webhook volume?" type: choice options: ["low (<100/min)", "medium (100-10K/min)", "high (>10K/min)"] # === DISTRIBUTION === canonical_source: "https://knowledgelib.io/software/patterns/webhook-implementation/2026" suggested_citation: "Source: knowledgelib.io — AI Knowledge Library (verified 2026-02-24)" # === RELATED UNITS === related_kos: depends_on: [] related_to: - id: software/patterns/idempotency-patterns/2026 label: "Idempotency Patterns" - id: software/patterns/retry-exponential-backoff/2026 label: "Retry with Exponential Backoff" - id: software/patterns/api-rate-limiting/2026 label: "API Rate Limiting" solves: - id: software/patterns/polling-sse-websocket/2026 label: "Polling vs SSE vs WebSocket — When to Use Each" alternative_to: - id: software/system-design/message-queue-event-driven/2026 label: "Message Queue & Event-Driven Architecture" often_confused_with: - id: software/system-design/webhooks-system/2026 label: "Webhooks System Design (architecture-level)" # === SOURCES (7 authoritative sources) === sources: - id: src1 title: "HMAC — Hash-based Message Authentication Code" author: webhooks.fyi url: https://webhooks.fyi/security/hmac type: community_resource published: 2024-01-15 reliability: high - id: src2 title: "Webhook Retry Best Practices" author: Svix url: https://www.svix.com/resources/webhook-best-practices/retries/ type: technical_blog published: 2024-06-10 reliability: high - id: src3 title: "Webhook Security Vulnerabilities Guide" author: Hookdeck url: https://hookdeck.com/webhooks/guides/webhook-security-vulnerabilities-guide type: technical_blog published: 2024-08-20 reliability: high - id: src4 title: "Implementing Webhook Retries" author: Hookdeck url: https://hookdeck.com/webhooks/guides/webhook-retry-best-practices type: technical_blog published: 2024-09-15 reliability: high - id: src5 title: "How to Implement SHA256 Webhook Signature Verification" author: Hookdeck url: https://hookdeck.com/webhooks/guides/how-to-implement-sha256-webhook-signature-verification type: technical_blog published: 2024-07-12 reliability: high - id: src6 title: "Design a Webhook System: Step-by-Step Guide" author: System Design Handbook url: https://www.systemdesignhandbook.com/guides/design-a-webhook-system/ type: technical_blog published: 2025-03-10 reliability: moderate_high - id: src7 title: "When to Use Webhooks, WebSocket, Pub/Sub, and Polling" author: Hookdeck url: https://hookdeck.com/webhooks/guides/when-to-use-webhooks type: technical_blog published: 2024-11-05 reliability: high --- # Webhook Implementation: Reliable Sender & Receiver Patterns ## TL;DR - **Bottom line**: A reliable webhook system requires HMAC-SHA256 signature verification on every payload, idempotency keys to handle at-least-once delivery, and exponential backoff with jitter for retries. - **Key tool/command**: `crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(received))` for constant-time signature verification. - **Watch out for**: Verifying HMAC against parsed/re-serialized JSON instead of the raw request body bytes -- this is the #1 cause of signature verification failures. - **Works with**: Any HTTP stack (Node.js, Python, Go, Java, Ruby). Language-agnostic pattern. HMAC-SHA256 used by 89% of webhook providers (Stripe, GitHub, Shopify, Slack, Okta). ## Constraints - Always sign payloads with HMAC-SHA256 using a per-endpoint secret -- unsigned webhooks are trivially spoofable - Always use constant-time comparison (`crypto.timingSafeEqual` / `hmac.compare_digest` / `hmac.Equal`) -- timing attacks leak secret bytes - Receivers must return 2xx within 5-10 seconds -- process asynchronously via queue, never synchronously - Exactly-once delivery is impossible over HTTP -- design for at-least-once with idempotency keys - Verify signatures against the raw request body bytes -- re-serialized JSON changes byte order and breaks HMAC - Include timestamps in signatures and reject payloads older than 5 minutes to prevent replay attacks ## Quick Reference | Component | Sender Responsibility | Receiver Responsibility | Security Consideration | |---|---|---|---| | Registration | Provide endpoint URL input + secret generation | Validate URL is HTTPS, store secret securely | Generate 256-bit secrets with CSPRNG | | Payload signing | HMAC-SHA256(secret, timestamp + "." + body) | Recompute HMAC from raw body + timestamp header | Sign raw bytes, never re-serialized JSON | | Signature header | Send as `X-Webhook-Signature` or `X-Hub-Signature-256` | Extract and compare with constant-time function | Use `crypto.timingSafeEqual` / `hmac.compare_digest` | | Timestamp | Include `X-Webhook-Timestamp` header (Unix epoch) | Reject if abs(now - timestamp) > 300 seconds | Prevents replay attacks | | Delivery | POST JSON with Content-Type + signature headers | Return 200/202 within 5 seconds | Use HTTPS only, verify TLS certificates | | Idempotency | Include unique `X-Webhook-Id` per event | Store processed IDs in DB/Redis (TTL 7-30 days) | Prevents duplicate processing from retries | | Retry logic | Exponential backoff: 1s, 2s, 4s, 8s... up to 12h | Return 2xx for success, 4xx for permanent failure | Add jitter to prevent thundering herd | | Dead letter queue | Move to DLQ after max retries (typically 5-8) | N/A (sender-side concern) | Alert on DLQ depth for monitoring | | Timeout | Set 10-30 second connection timeout | Enqueue work, respond fast | Sender retries on timeout, receiver must be idempotent | | Secret rotation | Support dual-secret window during rotation | Accept either old or new secret during transition | Rotate every 90 days minimum | ## Decision Tree ``` START ├── Are you the SENDER or RECEIVER? │ ├── SENDER ↓ │ │ ├── Scale > 10K events/min? │ │ │ ├── YES → Use persistent queue (Redis/SQS/RabbitMQ) + worker pool │ │ │ └── NO → In-process retry with backoff is sufficient │ │ ├── Need guaranteed ordering? │ │ │ ├── YES → Partition queue by endpoint ID, single consumer per partition │ │ │ └── NO → Parallel delivery workers (default, much higher throughput) │ │ └── DEFAULT → Queue-based sender with HMAC signing + exponential backoff │ │ │ └── RECEIVER ↓ │ ├── Processing takes > 5 seconds? │ │ ├── YES → ACK immediately (202), process via background queue │ │ └── NO → Process inline, return 200 with empty body │ ├── Duplicate events would cause harm (payments, orders)? │ │ ├── YES → Store webhook ID in DB with UNIQUE constraint before processing │ │ └── NO → Idempotency still recommended but less critical │ └── DEFAULT → Verify signature → check idempotency → enqueue → return 202 ``` ## Step-by-Step Guide ### 1. Generate and store webhook secrets Create a cryptographically secure random secret per registered endpoint. Never use predictable values. [src1] ```javascript // Node.js: generate a 256-bit webhook secret const crypto = require('crypto'); const secret = crypto.randomBytes(32).toString('hex'); // Store in database associated with the endpoint // Example: "a3f8b2c1d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0" ``` **Verify**: Secret is 64 hex characters (256 bits): `secret.length === 64` -> `true` ### 2. Sign webhook payloads (sender) Construct the signing input from the timestamp and raw body, then compute HMAC-SHA256. [src1] ```javascript // Node.js: sign a webhook payload const crypto = require('crypto'); function signWebhook(secret, timestamp, body) { const signingInput = `${timestamp}.${body}`; return crypto .createHmac('sha256', secret) .update(signingInput) .digest('hex'); } // Usage const timestamp = Math.floor(Date.now() / 1000).toString(); const body = JSON.stringify({ event: 'order.created', data: { id: 123 } }); const signature = signWebhook(secret, timestamp, body); ``` **Verify**: Signature is 64 hex characters: `signature.length === 64` -> `true` ### 3. Deliver with proper headers (sender) Send the webhook with signature, timestamp, and unique event ID headers. [src6] ```javascript // Node.js: deliver a webhook const https = require('https'); async function deliverWebhook(endpointUrl, body, secret) { const timestamp = Math.floor(Date.now() / 1000).toString(); const eventId = crypto.randomUUID(); const signature = signWebhook(secret, timestamp, body); const response = await fetch(endpointUrl, { method: 'POST', headers: { 'Content-Type': 'application/json', 'X-Webhook-Id': eventId, 'X-Webhook-Timestamp': timestamp, 'X-Webhook-Signature': `sha256=${signature}`, }, body: body, signal: AbortSignal.timeout(10000), // 10s timeout }); return { status: response.status, eventId }; } ``` **Verify**: Response status is 2xx: `response.status >= 200 && response.status < 300` ### 4. Verify signatures (receiver) Always verify against the raw body bytes using constant-time comparison. [src5] ```javascript // Node.js/Express: verify webhook signature const crypto = require('crypto'); function verifyWebhookSignature(secret, timestamp, rawBody, receivedSignature) { // Check timestamp is within 5 minutes const currentTime = Math.floor(Date.now() / 1000); if (Math.abs(currentTime - parseInt(timestamp, 10)) > 300) { throw new Error('Webhook timestamp too old — possible replay attack'); } // Recompute signature from raw body const signingInput = `${timestamp}.${rawBody}`; const expected = crypto .createHmac('sha256', secret) .update(signingInput) .digest('hex'); // Constant-time comparison const expectedBuf = Buffer.from(expected, 'utf8'); const receivedBuf = Buffer.from(receivedSignature.replace('sha256=', ''), 'utf8'); if (expectedBuf.length !== receivedBuf.length) { return false; } return crypto.timingSafeEqual(expectedBuf, receivedBuf); } ``` **Verify**: Returns `true` for a valid signature, `false` for tampered payloads ### 5. Implement idempotency (receiver) Store processed webhook IDs before processing to prevent duplicates. [src3] ```javascript // Idempotency check with database (PostgreSQL example) async function processWebhookIdempotent(db, webhookId, handler) { // Attempt to insert -- fails if already processed (UNIQUE constraint) try { await db.query( 'INSERT INTO processed_webhooks (webhook_id, received_at) VALUES ($1, NOW())', [webhookId] ); } catch (err) { if (err.code === '23505') { // unique_violation return { status: 'duplicate', message: 'Already processed' }; } throw err; } // Process the webhook const result = await handler(); await db.query( 'UPDATE processed_webhooks SET processed_at = NOW(), status = $1 WHERE webhook_id = $2', ['completed', webhookId] ); return result; } ``` **Verify**: Sending the same webhook ID twice results in `{ status: 'duplicate' }` on the second call ### 6. Add retry with exponential backoff and jitter (sender) Retry failed deliveries with increasing delays and randomized jitter to avoid thundering herd. [src2] ```javascript // Retry with exponential backoff + full jitter async function deliverWithRetry(endpointUrl, body, secret, maxRetries = 5) { for (let attempt = 0; attempt <= maxRetries; attempt++) { try { const result = await deliverWebhook(endpointUrl, body, secret); if (result.status >= 200 && result.status < 300) { return { success: true, attempt, eventId: result.eventId }; } if (result.status >= 400 && result.status < 500) { // 4xx = permanent failure, don't retry (except 429) if (result.status !== 429) { return { success: false, attempt, status: result.status, permanent: true }; } } } catch (err) { // Network error or timeout — will retry } if (attempt < maxRetries) { const baseDelay = Math.min(1000 * Math.pow(2, attempt), 43200000); // cap at 12 hours const jitter = Math.random() * baseDelay; // full jitter await new Promise(resolve => setTimeout(resolve, jitter)); } } return { success: false, exhausted: true }; // Move to dead letter queue } ``` **Verify**: Failed delivery retries with increasing delays: attempt 0 = immediate, attempt 1 = 0-2s, attempt 2 = 0-4s, etc. ## Code Examples ### Node.js: Complete Webhook Receiver (Express) ```javascript // Input: POST request with JSON body + signature headers // Output: 200/202 for valid webhooks, 401 for invalid signature const express = require('express'); const crypto = require('crypto'); const app = express(); // CRITICAL: capture raw body for HMAC verification app.use(express.json({ verify: (req, res, buf) => { req.rawBody = buf.toString('utf8'); } })); app.post('/webhooks', (req, res) => { const signature = req.headers['x-webhook-signature']; const timestamp = req.headers['x-webhook-timestamp']; const webhookId = req.headers['x-webhook-id']; if (!signature || !timestamp || !webhookId) { return res.status(401).json({ error: 'Missing headers' }); } const isValid = verifyWebhookSignature( process.env.WEBHOOK_SECRET, timestamp, req.rawBody, signature ); if (!isValid) return res.status(401).json({ error: 'Invalid signature' }); // Enqueue for async processing, respond immediately queue.add({ webhookId, body: req.body }); res.status(202).json({ received: true }); }); ``` ### Python: Webhook Signature Verification (Flask) > Full script: [python-webhook-signature-verification-flask.py](scripts/python-webhook-signature-verification-flask.py) (28 lines) ```python # Input: POST request with JSON body + HMAC signature headers # Output: 200/202 for valid webhooks, 401 for invalid import hmac import hashlib import time # ... (see full script) ``` ### Go: Webhook Signature Verification > Full script: [go-webhook-signature-verification.go](scripts/go-webhook-signature-verification.go) (39 lines) ```go // Input: HTTP POST with JSON body + HMAC signature headers // Output: 200/202 for valid webhooks, 401 for invalid package main import ( "crypto/hmac" # ... (see full script) ``` ## Anti-Patterns ### Wrong: Verifying HMAC against parsed JSON ```javascript // BAD -- re-serialized JSON changes key order and whitespace app.post('/webhooks', express.json(), (req, res) => { const body = JSON.stringify(req.body); // key order may differ! const expected = crypto.createHmac('sha256', secret) .update(body).digest('hex'); if (expected === receivedSignature) { /* ... */ } }); ``` ### Correct: Verifying against raw body bytes ```javascript // GOOD -- use the exact bytes the sender signed app.use(express.json({ verify: (req, res, buf) => { req.rawBody = buf.toString('utf8'); } })); app.post('/webhooks', (req, res) => { const expected = crypto.createHmac('sha256', secret) .update(req.rawBody).digest('hex'); // ... }); ``` ### Wrong: Using equality operator for signature comparison ```javascript // BAD -- string === leaks timing information if (computedSignature === receivedSignature) { processWebhook(req.body); } ``` ### Correct: Using constant-time comparison ```javascript // GOOD -- constant-time comparison prevents timing attacks const expected = Buffer.from(computedSignature, 'hex'); const received = Buffer.from(receivedSignature, 'hex'); if (expected.length === received.length && crypto.timingSafeEqual(expected, received)) { processWebhook(req.body); } ``` ### Wrong: Processing webhooks synchronously before responding ```javascript // BAD -- long processing causes sender timeout and unnecessary retries app.post('/webhooks', async (req, res) => { await updateDatabase(req.body); // 2 seconds await sendNotification(req.body); // 3 seconds await generateReport(req.body); // 5 seconds res.status(200).json({ ok: true }); // 10 seconds total -- sender may have timed out }); ``` ### Correct: Acknowledge fast, process asynchronously ```javascript // GOOD -- respond immediately, process in background app.post('/webhooks', (req, res) => { queue.add({ event: req.body, webhookId: req.headers['x-webhook-id'] }); res.status(202).json({ received: true }); // < 50ms response }); ``` ### Wrong: No dead letter queue ```javascript // BAD -- retries forever, wasting resources on permanently failed endpoints async function deliver(url, body) { while (true) { try { await fetch(url, { method: 'POST', body }); return; } catch { await sleep(1000); } // infinite retry loop } } ``` ### Correct: Bounded retries with dead letter queue ```javascript // GOOD -- cap retries, move failures to DLQ for investigation async function deliver(url, body, maxRetries = 5) { for (let i = 0; i <= maxRetries; i++) { try { const res = await fetch(url, { method: 'POST', body }); if (res.ok) return { delivered: true }; } catch {} const delay = Math.min(1000 * 2 ** i, 43200000) * Math.random(); await sleep(delay); } await deadLetterQueue.add({ url, body, failedAt: new Date() }); return { delivered: false, dlq: true }; } ``` ## Common Pitfalls - **Signature fails after JSON middleware**: Body parsers consume the raw stream. Use `express.json({ verify })` or `request.get_data()` in Flask to capture raw bytes before parsing. Fix: add `verify` callback in Express or read raw body first. [src5] - **Replay attacks on endpoints without timestamp check**: Without timestamp validation, a captured webhook can be replayed indefinitely. Fix: include timestamp in signing input and reject if `|now - timestamp| > 300`. [src3] - **Thundering herd on retry**: When many webhooks fail simultaneously (endpoint downtime), synchronized retries overload the recovering endpoint. Fix: add full jitter (`Math.random() * baseDelay`) to exponential backoff. [src2] - **No idempotency key on payment/order webhooks**: At-least-once delivery means duplicates will happen. Without idempotency, a retried `payment.completed` event charges the customer twice. Fix: store `webhook_id` with UNIQUE constraint before processing. [src3] - **Ignoring 4xx vs 5xx distinction**: Retrying on 400 Bad Request wastes resources -- the payload is permanently invalid. Fix: only retry on 5xx and network errors; treat 4xx (except 429) as permanent failures. [src4] - **Webhook secret in client-side code**: Exposing the HMAC secret allows forging signatures. Fix: keep secrets server-side only, use environment variables. [src1] - **Clock drift between sender and receiver**: Strict timestamp windows fail when server clocks diverge. Fix: use NTP on both sides, set 5-minute window (not 30 seconds). [src1] ## Diagnostic Commands ```bash # Test webhook delivery locally (sender simulation) curl -X POST http://localhost:3000/webhooks \ -H "Content-Type: application/json" \ -H "X-Webhook-Signature: sha256=$(echo -n '1708800000.{"event":"test"}' | openssl dgst -sha256 -hmac 'your-secret' | awk '{print $2}')" \ -H "X-Webhook-Timestamp: 1708800000" \ -H "X-Webhook-Id: test-$(uuidgen)" \ -d '{"event":"test"}' # Compute HMAC-SHA256 signature manually echo -n 'timestamp.payload' | openssl dgst -sha256 -hmac 'your-secret-key' # Test signature verification in Python python3 -c "import hmac, hashlib; print(hmac.new(b'secret', b'1708800000.{\"test\":1}', hashlib.sha256).hexdigest())" # Monitor webhook delivery queue depth (Redis) redis-cli LLEN webhook:delivery:queue # Check dead letter queue redis-cli LLEN webhook:dlq # Inspect recent failed deliveries (PostgreSQL) psql -c "SELECT endpoint_url, status_code, attempts, last_error FROM webhook_deliveries WHERE status = 'failed' ORDER BY updated_at DESC LIMIT 20;" ``` ## When to Use / When Not to Use | Use When | Don't Use When | Use Instead | |---|---|---| | Server-to-server event notification (backend-to-backend) | Client needs real-time updates from server | Server-Sent Events (SSE) or WebSockets | | Decoupled systems where receiver controls the endpoint | Guaranteed strict event ordering is required | Message queue (Kafka, RabbitMQ, SQS) | | Low-to-medium frequency events (<10K/min per endpoint) | Ultra-high throughput (>100K events/sec) | Event streaming (Kafka, Pulsar) | | Receiver is a third-party system you don't control | Both sides are within the same infrastructure | Internal message queue or pub/sub | | Simple integration with no persistent connection needed | Bidirectional communication is required | WebSockets | | Serverless/edge receivers (can't hold connections open) | Receiver is frequently offline for extended periods | Message queue with durable storage | | Push model preferred over pull (reduce polling overhead) | Minutes-old data is acceptable | Simple polling with caching | ## Important Caveats - HMAC-SHA256 is the dominant standard (89% of providers), but some use Ed25519 (Svix) or RSA-SHA256 -- always check the provider's documentation for header names and signing scheme - The `Standard Webhooks` specification (standardwebhooks.com) is gaining adoption but is not yet universally implemented -- Svix, Clerk, and Resend use it - At high scale (>10K events/min), in-process retry loops become a bottleneck; use a dedicated queue (Redis, SQS, RabbitMQ) with worker processes - Webhook receivers behind CDNs or WAFs may have request body size limits or header filtering that silently breaks signature verification - Some providers (Stripe, GitHub) use different header names (`Stripe-Signature`, `X-Hub-Signature-256`) -- there is no universal standard header name - IP allowlisting provides defense-in-depth but is insufficient alone -- always verify HMAC signatures regardless of source IP ## Related Units - [Idempotency Patterns](/software/patterns/idempotency-patterns/2026) - [Retry with Exponential Backoff](/software/patterns/retry-exponential-backoff/2026) - [API Rate Limiting](/software/patterns/api-rate-limiting/2026) - [Polling vs SSE vs WebSocket](/software/patterns/polling-sse-websocket/2026) - [Webhooks System Design](/software/system-design/webhooks-system/2026)