Webhook Implementation: Reliable Sender & Receiver Patterns

TL;DR

• Bottom line: A reliable webhook system requires HMAC-SHA256 signature verification on every payload, idempotency keys to handle at-least-once delivery, and exponential backoff with jitter for retries.
• Key tool/command: crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(received)) for constant-time signature verification.
• Watch out for: Verifying HMAC against parsed/re-serialized JSON instead of the raw request body bytes — the #1 cause of signature verification failures.
• Works with: Any HTTP stack (Node.js, Python, Go, Java, Ruby). Language-agnostic pattern. HMAC-SHA256 used by 89% of webhook providers.

Constraints

• Always sign payloads with HMAC-SHA256 using a per-endpoint secret — unsigned webhooks are trivially spoofable
• Always use constant-time comparison (crypto.timingSafeEqual / hmac.compare_digest / hmac.Equal) — timing attacks leak secret bytes
• Receivers must return 2xx within 5-10 seconds — process asynchronously via queue, never synchronously
• Exactly-once delivery is impossible over HTTP — design for at-least-once with idempotency keys
• Verify signatures against the raw request body bytes — re-serialized JSON changes byte order and breaks HMAC
• Include timestamps in signatures and reject payloads older than 5 minutes to prevent replay attacks

Quick Reference

Decision Tree

START
├── Are you the SENDER or RECEIVER?
│   ├── SENDER ↓
│   │   ├── Scale > 10K events/min?
│   │   │   ├── YES → Use persistent queue (Redis/SQS/RabbitMQ) + worker pool
│   │   │   └── NO → In-process retry with backoff is sufficient
│   │   ├── Need guaranteed ordering?
│   │   │   ├── YES → Partition queue by endpoint ID, single consumer per partition
│   │   │   └── NO → Parallel delivery workers (default, much higher throughput)
│   │   └── DEFAULT → Queue-based sender with HMAC signing + exponential backoff
│   │
│   └── RECEIVER ↓
│       ├── Processing takes > 5 seconds?
│       │   ├── YES → ACK immediately (202), process via background queue
│       │   └── NO → Process inline, return 200 with empty body
│       ├── Duplicate events would cause harm (payments, orders)?
│       │   ├── YES → Store webhook ID in DB with UNIQUE constraint before processing
│       │   └── NO → Idempotency still recommended but less critical
│       └── DEFAULT → Verify signature → check idempotency → enqueue → return 202

Step-by-Step Guide

1. Generate and store webhook secrets

Create a cryptographically secure random secret per registered endpoint. Never use predictable values. [src1]

const crypto = require('crypto');
const secret = crypto.randomBytes(32).toString('hex');
// Store in database associated with the endpoint

Verify: Secret is 64 hex characters (256 bits): secret.length === 64 → true

2. Sign webhook payloads (sender)

Construct the signing input from the timestamp and raw body, then compute HMAC-SHA256. [src1]

function signWebhook(secret, timestamp, body) {
  const signingInput = `${timestamp}.${body}`;
  return crypto
    .createHmac('sha256', secret)
    .update(signingInput)
    .digest('hex');
}

Verify: Signature is 64 hex characters: signature.length === 64 → true

3. Deliver with proper headers (sender)

Send the webhook with signature, timestamp, and unique event ID headers. [src6]

async function deliverWebhook(endpointUrl, body, secret) {
  const timestamp = Math.floor(Date.now() / 1000).toString();
  const eventId = crypto.randomUUID();
  const signature = signWebhook(secret, timestamp, body);
  const response = await fetch(endpointUrl, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'X-Webhook-Id': eventId,
      'X-Webhook-Timestamp': timestamp,
      'X-Webhook-Signature': `sha256=${signature}`,
    },
    body: body,
    signal: AbortSignal.timeout(10000),
  });
  return { status: response.status, eventId };
}

Verify: Response status is 2xx: response.status >= 200 && response.status < 300

4. Verify signatures (receiver)

Always verify against the raw body bytes using constant-time comparison. [src5]

function verifyWebhookSignature(secret, timestamp, rawBody, receivedSignature) {
  const currentTime = Math.floor(Date.now() / 1000);
  if (Math.abs(currentTime - parseInt(timestamp, 10)) > 300) {
    throw new Error('Webhook timestamp too old');
  }
  const signingInput = `${timestamp}.${rawBody}`;
  const expected = crypto.createHmac('sha256', secret)
    .update(signingInput).digest('hex');
  const expectedBuf = Buffer.from(expected, 'utf8');
  const receivedBuf = Buffer.from(
    receivedSignature.replace('sha256=', ''), 'utf8'
  );
  if (expectedBuf.length !== receivedBuf.length) return false;
  return crypto.timingSafeEqual(expectedBuf, receivedBuf);
}

Verify: Returns true for valid signatures, false for tampered payloads

5. Implement idempotency (receiver)

Store processed webhook IDs before processing to prevent duplicates. [src3]

async function processWebhookIdempotent(db, webhookId, handler) {
  try {
    await db.query(
      'INSERT INTO processed_webhooks (webhook_id, received_at) VALUES ($1, NOW())',
      [webhookId]
    );
  } catch (err) {
    if (err.code === '23505') { // unique_violation
      return { status: 'duplicate', message: 'Already processed' };
    }
    throw err;
  }
  const result = await handler();
  return result;
}

Verify: Sending the same webhook ID twice returns { status: 'duplicate' } on second call

6. Add retry with exponential backoff and jitter (sender)

Retry failed deliveries with increasing delays and randomized jitter. [src2]

async function deliverWithRetry(endpointUrl, body, secret, maxRetries = 5) {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      const result = await deliverWebhook(endpointUrl, body, secret);
      if (result.status >= 200 && result.status < 300) {
        return { success: true, attempt, eventId: result.eventId };
      }
      if (result.status >= 400 && result.status < 500 && result.status !== 429) {
        return { success: false, permanent: true };
      }
    } catch (err) { /* network error — retry */ }
    if (attempt < maxRetries) {
      const baseDelay = Math.min(1000 * Math.pow(2, attempt), 43200000);
      await new Promise(r => setTimeout(r, Math.random() * baseDelay));
    }
  }
  return { success: false, exhausted: true };
}

Verify: Failed delivery retries with increasing delays and moves to DLQ after max retries

Code Examples

Node.js/Express: Complete Webhook Receiver

// Input:  POST request with JSON body + signature headers
// Output: 200/202 for valid webhooks, 401 for invalid signature

const express = require('express');
const crypto = require('crypto');
const app = express();

app.use(express.json({
  verify: (req, res, buf) => { req.rawBody = buf.toString('utf8'); }
}));

app.post('/webhooks', (req, res) => {
  const signature = req.headers['x-webhook-signature'];
  const timestamp = req.headers['x-webhook-timestamp'];
  const webhookId = req.headers['x-webhook-id'];

  if (!signature || !timestamp || !webhookId) {
    return res.status(401).json({ error: 'Missing headers' });
  }

  const isValid = verifyWebhookSignature(
    process.env.WEBHOOK_SECRET, timestamp, req.rawBody, signature
  );
  if (!isValid) return res.status(401).json({ error: 'Invalid signature' });

  queue.add({ webhookId, body: req.body });
  res.status(202).json({ received: true });
});

Python/Flask: Webhook Signature Verification

# Input:  POST request with JSON body + HMAC signature headers
# Output: 200/202 for valid webhooks, 401 for invalid

import hmac, hashlib, time
from flask import Flask, request, jsonify

app = Flask(__name__)

def verify_signature(secret, timestamp, raw_body, received_sig):
    if abs(time.time() - int(timestamp)) > 300:
        return False
    signing_input = f"{timestamp}.".encode() + raw_body
    expected = hmac.new(
        secret.encode(), signing_input, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, received_sig.removeprefix("sha256="))

@app.route("/webhooks", methods=["POST"])
def receive_webhook():
    sig = request.headers.get("X-Webhook-Signature", "")
    ts = request.headers.get("X-Webhook-Timestamp", "")
    raw = request.get_data()  # raw bytes, NOT request.json
    if not verify_signature(app.config["WEBHOOK_SECRET"], ts, raw, sig):
        return jsonify({"error": "Invalid signature"}), 401
    task_queue.enqueue(process_event, request.json)
    return jsonify({"received": True}), 202

Go: Webhook Signature Verification

// Input:  HTTP POST with JSON body + HMAC signature headers
// Output: 200/202 for valid webhooks, 401 for invalid

func verifySignature(secret, timestamp string, rawBody []byte, receivedSig string) bool {
    ts, err := strconv.ParseInt(timestamp, 10, 64)
    if err != nil || math.Abs(float64(time.Now().Unix()-ts)) > 300 {
        return false
    }
    signingInput := fmt.Sprintf("%s.%s", timestamp, string(rawBody))
    mac := hmac.New(sha256.New, []byte(secret))
    mac.Write([]byte(signingInput))
    expected := hex.EncodeToString(mac.Sum(nil))
    return hmac.Equal([]byte(expected), []byte(receivedSig))
}

func webhookHandler(w http.ResponseWriter, r *http.Request) {
    rawBody, _ := io.ReadAll(r.Body)
    sig := r.Header.Get("X-Webhook-Signature")
    ts := r.Header.Get("X-Webhook-Timestamp")
    if !verifySignature(os.Getenv("WEBHOOK_SECRET"), ts, rawBody, sig) {
        http.Error(w, "Invalid signature", http.StatusUnauthorized)
        return
    }
    w.WriteHeader(http.StatusAccepted)
    w.Write([]byte(`{"received":true}`))
}

Anti-Patterns

Wrong: Verifying HMAC against parsed JSON

// BAD — re-serialized JSON changes key order and whitespace
app.post('/webhooks', express.json(), (req, res) => {
  const body = JSON.stringify(req.body); // key order may differ!
  const expected = crypto.createHmac('sha256', secret)
    .update(body).digest('hex');
  if (expected === receivedSignature) { /* ... */ }
});

Correct: Verifying against raw body bytes

// GOOD — use the exact bytes the sender signed
app.use(express.json({
  verify: (req, res, buf) => { req.rawBody = buf.toString('utf8'); }
}));
app.post('/webhooks', (req, res) => {
  const expected = crypto.createHmac('sha256', secret)
    .update(req.rawBody).digest('hex');
  // constant-time comparison follows...
});

Wrong: Using equality operator for signature comparison

// BAD — string === leaks timing information
if (computedSignature === receivedSignature) {
  processWebhook(req.body);
}

Correct: Using constant-time comparison

// GOOD — constant-time comparison prevents timing attacks
const expected = Buffer.from(computedSignature, 'hex');
const received = Buffer.from(receivedSignature, 'hex');
if (expected.length === received.length &&
    crypto.timingSafeEqual(expected, received)) {
  processWebhook(req.body);
}

Wrong: Processing webhooks synchronously before responding

// BAD — long processing causes sender timeout and unnecessary retries
app.post('/webhooks', async (req, res) => {
  await updateDatabase(req.body);       // 2 seconds
  await sendNotification(req.body);     // 3 seconds
  await generateReport(req.body);       // 5 seconds
  res.status(200).json({ ok: true });   // 10s total — sender may timeout
});

Correct: Acknowledge fast, process asynchronously

// GOOD — respond immediately, process in background
app.post('/webhooks', (req, res) => {
  queue.add({ event: req.body, webhookId: req.headers['x-webhook-id'] });
  res.status(202).json({ received: true }); // < 50ms response
});

Wrong: No dead letter queue

// BAD — retries forever, wasting resources
async function deliver(url, body) {
  while (true) {
    try { await fetch(url, { method: 'POST', body }); return; }
    catch { await sleep(1000); }
  }
}

Correct: Bounded retries with dead letter queue

// GOOD — cap retries, move failures to DLQ
async function deliver(url, body, maxRetries = 5) {
  for (let i = 0; i <= maxRetries; i++) {
    try {
      const res = await fetch(url, { method: 'POST', body });
      if (res.ok) return { delivered: true };
    } catch {}
    const delay = Math.min(1000 * 2 ** i, 43200000) * Math.random();
    await sleep(delay);
  }
  await deadLetterQueue.add({ url, body, failedAt: new Date() });
  return { delivered: false, dlq: true };
}

Common Pitfalls

• Signature fails after JSON middleware: Body parsers consume the raw stream. Use express.json({ verify }) or request.get_data() in Flask to capture raw bytes before parsing. [src5]
• Replay attacks on endpoints without timestamp check: Without timestamp validation, captured webhooks can be replayed indefinitely. Fix: include timestamp in signing input and reject if |now - timestamp| > 300s. [src3]
• Thundering herd on retry: Synchronized retries after mass failure overload recovering endpoints. Fix: add full jitter (Math.random() * baseDelay) to exponential backoff. [src2]
• No idempotency key on payment/order webhooks: At-least-once delivery means duplicates will happen. Fix: store webhook_id with UNIQUE constraint before processing. [src3]
• Retrying on 4xx errors: A 400 Bad Request is permanent — the payload is invalid. Fix: only retry on 5xx and network errors; treat 4xx (except 429) as permanent failures. [src4]
• Webhook secret in client-side code: Exposing the HMAC secret allows forging signatures. Fix: keep secrets server-side only, use environment variables. [src1]
• Clock drift between sender and receiver: Strict timestamp windows fail when server clocks diverge. Fix: use NTP on both sides, set 5-minute window. [src1]

Diagnostic Commands

# Test webhook delivery locally (sender simulation)
curl -X POST http://localhost:3000/webhooks \
  -H "Content-Type: application/json" \
  -H "X-Webhook-Signature: sha256=$(echo -n '1708800000.{\"event\":\"test\"}' | openssl dgst -sha256 -hmac 'your-secret' | awk '{print $2}')" \
  -H "X-Webhook-Timestamp: 1708800000" \
  -H "X-Webhook-Id: test-$(uuidgen)" \
  -d '{"event":"test"}'

# Compute HMAC-SHA256 signature manually
echo -n 'timestamp.payload' | openssl dgst -sha256 -hmac 'your-secret-key'

# Test signature verification in Python
python3 -c "import hmac, hashlib; print(hmac.new(b'secret', b'1708800000.{\"test\":1}', hashlib.sha256).hexdigest())"

# Monitor webhook delivery queue depth (Redis)
redis-cli LLEN webhook:delivery:queue

# Check dead letter queue
redis-cli LLEN webhook:dlq

When to Use / When Not to Use

Important Caveats

• HMAC-SHA256 is the dominant standard (89% of providers), but some use Ed25519 (Svix) or RSA-SHA256 — always check provider docs
• The Standard Webhooks specification (standardwebhooks.com) is gaining adoption but is not yet universally implemented
• At high scale (>10K events/min), in-process retry loops become a bottleneck — use a dedicated queue (Redis, SQS, RabbitMQ)
• Webhook receivers behind CDNs or WAFs may have body size limits or header filtering that breaks signature verification
• Some providers use different header names (Stripe-Signature, X-Hub-Signature-256) — there is no universal standard header name
• IP allowlisting provides defense-in-depth but is insufficient alone — always verify HMAC signatures regardless of source IP