---
# === IDENTITY ===
id: software/system-design/notification-system/2026
canonical_question: "How do I design a scalable notification system (push, in-app, email)?"
aliases:
  - "notification service architecture"
  - "push notification system design"
  - "multi-channel notification platform"
  - "real-time notification infrastructure"
entity_type: software_reference
domain: software > system-design > notification_system
region: global
jurisdiction: global
temporal_scope: 2020-2026

# === VERIFICATION ===
last_verified: 2026-02-23
confidence: 0.92
version: 1.0
first_published: 2026-02-23

# === TEMPORAL VALIDITY ===
temporal_validity:
  status: stable
  last_breaking_change: "FCM HTTP v1 API replaced legacy API (June 2024)"
  next_review: 2026-08-22
  change_sensitivity: medium

# === CONSTRAINTS ===
constraints:
  - "Always check user notification preferences and opt-out status before sending any notification"
  - "Use separate message queues per channel (email, push, SMS, in-app) to prevent one channel failure from blocking others"
  - "Never send push notifications without valid device tokens; stale tokens must be pruned on 410/InvalidRegistration responses"
  - "FCM legacy HTTP and XMPP APIs were removed June 2024; use HTTP v1 API only"
  - "Email notifications must comply with CAN-SPAM (US), CASL (Canada), and GDPR (EU) opt-in/opt-out requirements"

# === SKIP CONDITIONS ===
skip_this_unit_if:
  - condition: "Need to design a chat/messaging system with bidirectional real-time communication"
    use_instead: "software/system-design/chat-system/2026"
  - condition: "Need to build an event-driven microservices architecture (not notification-specific)"
    use_instead: "software/system-design/event-driven-architecture/2026"
  - condition: "Need email deliverability optimization only (SPF/DKIM/DMARC)"
    use_instead: "software/system-design/email-deliverability/2026"

# === AGENT HINTS ===
inputs_needed:
  - key: scale
    question: "What is your expected notification volume?"
    type: choice
    options: ["<1K/day", "1K-100K/day", "100K-10M/day", ">10M/day"]
  - key: channels
    question: "Which notification channels do you need?"
    type: choice
    options: ["push-only", "email-only", "push+email", "push+email+in-app", "all (push+email+SMS+in-app)"]
  - key: latency
    question: "What is the acceptable delivery latency for critical notifications?"
    type: choice
    options: ["<1s (real-time)", "<30s (near-real-time)", "<5min (batch-acceptable)"]

# === DISTRIBUTION ===
canonical_source: "https://knowledgelib.io/software/system-design/notification-system/2026"
suggested_citation: "Source: knowledgelib.io -- AI Knowledge Library (verified 2026-02-23)"

# === RELATED UNITS ===
related_kos:
  depends_on:
    - id: "software/system-design/message-queues/2026"
      label: "Message Queue Architecture (Kafka vs RabbitMQ vs SQS)"
  related_to:
    - id: "software/system-design/rate-limiting/2026"
      label: "Rate Limiting and Throttling Patterns"
    - id: "software/system-design/event-driven-architecture/2026"
      label: "Event-Driven Architecture"
  solves:
    - id: "software/debugging/notification-delivery-failures/2026"
      label: "Debugging Notification Delivery Failures"
  alternative_to:
    - id: "software/system-design/webhook-system/2026"
      label: "Webhook Delivery System Design"
  often_confused_with:
    - id: "software/system-design/chat-system/2026"
      label: "Real-Time Chat System Design"

# === SOURCES (7 authoritative sources) ===
sources:
  - id: src1
    title: "FCM Architectural Overview"
    author: Google Firebase
    url: https://firebase.google.com/docs/cloud-messaging/fcm-architecture
    type: official_docs
    published: 2024-06-01
    reliability: authoritative
  - id: src2
    title: "Sending Notification Requests to APNs"
    author: Apple Developer
    url: https://developer.apple.com/documentation/usernotifications/sending-notification-requests-to-apns
    type: official_docs
    published: 2024-09-01
    reliability: authoritative
  - id: src3
    title: "Design a Scalable Notification Service"
    author: AlgoMaster
    url: https://blog.algomaster.io/p/design-a-scalable-notification-service
    type: technical_blog
    published: 2024-11-15
    reliability: high
  - id: src4
    title: "How to Design a Notification System: A Complete Guide"
    author: System Design Handbook
    url: https://www.systemdesignhandbook.com/guides/design-a-notification-system/
    type: technical_blog
    published: 2024-08-20
    reliability: high
  - id: src5
    title: "Architecting a Scalable Notification System (Push, Email, SMS, In-App)"
    author: Meerako
    url: https://www.meerako.com/blogs/building-scalable-notification-system-architecture-aws
    type: technical_blog
    published: 2024-10-05
    reliability: moderate_high
  - id: src6
    title: "Stable Architecture and Successful Email Sending at Scale"
    author: SendGrid (Twilio)
    url: https://sendgrid.com/en-us/blog/stable-architecture-and-successful-email-sending-at-scale
    type: technical_blog
    published: 2023-03-12
    reliability: high
  - id: src7
    title: "Notification System Design: Architecture & Best Practices"
    author: MagicBell
    url: https://www.magicbell.com/blog/notification-system-design
    type: technical_blog
    published: 2024-07-18
    reliability: moderate_high
---

# Scalable Notification System Design (Push, In-App, Email)

## TL;DR

- **Bottom line**: A scalable notification system uses event-driven fan-out with per-channel message queues, a user preference service, dedicated channel processors (push/email/SMS/in-app), and a retry-with-DLQ strategy to reliably deliver millions of notifications daily.
- **Key tool/command**: `SNS -> SQS -> Lambda/Worker` fan-out pattern (or Kafka topics with consumer groups for higher throughput)
- **Watch out for**: Sending notifications without checking user preferences first -- this causes opt-out violations, spam complaints, and CAN-SPAM/GDPR fines.
- **Works with**: FCM HTTP v1 API (Android/web), APNs HTTP/2 (iOS), SendGrid/SES (email), WebSocket/SSE (in-app). Cloud-agnostic architecture; AWS, GCP, Azure all supported.

## Constraints
<!-- Agents: read this section before recommending any action from this unit.
     These are hard constraints that must not be violated. -->

- Always query the user preference service and check opt-in status before dispatching any notification to any channel
- Use separate message queues per channel so a failure in email delivery does not block push or in-app notifications
- FCM legacy HTTP API and XMPP API were removed in June 2024; all push to Android/web must use HTTP v1 API
- APNs requires HTTP/2 connections on port 443 (or 2197); use token-based authentication (keys do not expire, unlike certificates)
- Email notifications must include a one-click unsubscribe header (RFC 8058) and physical mailing address per CAN-SPAM
- Never store raw device tokens or APNs certificates in client-side code or public repositories

## Quick Reference

| Component | Role | Technology Options | Scaling Strategy |
|---|---|---|---|
| API Gateway | Rate-limit, authenticate, route notification requests | Kong, AWS API Gateway, Nginx | Horizontal + auto-scale on request rate |
| Notification Service | Validate payload, enrich with user prefs, fan-out to queues | Node.js, Go, Java Spring Boot | Stateless; scale by CPU/request count |
| User Preference Service | Store per-user channel opt-ins, quiet hours, frequency caps | PostgreSQL + Redis cache | Read-heavy; cache in Redis with 5-min TTL |
| Template Engine | Render channel-specific content from templates | Handlebars, Jinja2, MJML (email) | Stateless; co-locate with notification service |
| Message Queue | Decouple submission from delivery, buffer spikes | Kafka, RabbitMQ, AWS SQS+SNS | Partition by channel; scale consumers independently |
| Priority Router | Classify notifications (critical/high/normal/low) and route to priority queues | Custom logic + separate queue topics | Separate queues per priority tier |
| Push Processor | Deliver to mobile/web via FCM and APNs | FCM HTTP v1 API, APNs HTTP/2 | Batch sends (FCM: 500 tokens/multicast); scale workers by queue depth |
| Email Processor | Render and send email via ESP | SendGrid, Amazon SES, Mailgun | Warm up IPs; separate transactional vs marketing subdomains |
| SMS Processor | Send SMS via telecom API | Twilio, Nexmo/Vonage, AWS SNS | Rate-limit per country; scale by queue depth |
| In-App Processor | Deliver real-time to connected clients | WebSocket (Socket.io), SSE, long polling | Sticky sessions or Redis Pub/Sub for horizontal scale |
| Notification Store | Persist notification history and read/unread status | PostgreSQL (recent), S3/cold storage (archive) | Time-partition; archive after 90 days |
| Retry / DLQ | Handle failed deliveries with exponential backoff | Built-in queue retry + Dead Letter Queue | Alert on DLQ depth; manual inspection workflow |
| Analytics & Monitoring | Track delivery rates, open rates, latency | Prometheus + Grafana, ELK Stack, Datadog | Aggregate metrics; alert on delivery rate drops |
| Device Registry | Store and validate device tokens per user | PostgreSQL or DynamoDB | Prune invalid tokens on provider feedback |

## Decision Tree

> Full script: [decision-tree.txt](scripts/decision-tree.txt) (38 lines)

```
START: What notification channels do you need?
|
+-- Push notifications only?
|   +-- YES --> Use FCM (Android/web) + APNs (iOS) directly
|   |           with a simple queue (SQS/Redis) in front
# ... (see full script)
```

## Step-by-Step Guide

### 1. Define notification events and payload schema

Establish a canonical event format that all services publish. Include event type, recipient, channel hints, priority, and idempotency key. [src3]

```json
{
  "event_id": "evt_abc123",
  "event_type": "order.shipped",
  "recipient_id": "user_789",
  "channels": ["push", "email", "in_app"],
  "priority": "high",
  "data": {
    "order_id": "ORD-456",
    "tracking_url": "https://example.com/track/ORD-456"
  },
  "idempotency_key": "order_shipped_ORD-456",
  "created_at": "2026-02-23T10:00:00Z"
}
```

**Verify**: Validate schema with JSON Schema or Zod. All downstream processors must accept this format without transformation.

### 2. Build the user preference service

Store per-user, per-channel, per-notification-type preferences. Cache aggressively since this is read on every notification. [src4]

```sql
CREATE TABLE user_notification_preferences (
  user_id       UUID NOT NULL,
  channel       TEXT NOT NULL CHECK (channel IN ('push','email','sms','in_app')),
  notif_type    TEXT NOT NULL,  -- e.g. 'order_updates', 'marketing', 'security'
  enabled       BOOLEAN DEFAULT true,
  quiet_start   TIME,          -- e.g. 22:00
  quiet_end     TIME,          -- e.g. 08:00
  timezone      TEXT DEFAULT 'UTC',
  updated_at    TIMESTAMPTZ DEFAULT now(),
  PRIMARY KEY (user_id, channel, notif_type)
);
```

**Verify**: `SELECT * FROM user_notification_preferences WHERE user_id = 'test-user';` --> should return one row per channel/type combination.

### 3. Implement fan-out with message queues

Publish the notification event to a central topic. Subscribers (one per channel) receive a copy and enqueue it for processing. [src5]

```python
# AWS SNS + SQS fan-out example
import boto3
import json

sns = boto3.client('sns', region_name='us-east-1')
TOPIC_ARN = 'arn:aws:sns:us-east-1:123456789:notifications'

def publish_notification(event: dict):
    """Publish notification event to SNS for fan-out to channel queues."""
    sns.publish(
        TopicArn=TOPIC_ARN,
        Message=json.dumps(event),
        MessageAttributes={
            'priority': {
                'DataType': 'String',
                'StringValue': event.get('priority', 'normal')
            }
        }
    )
```

**Verify**: Check SQS queue depth after publishing: `aws sqs get-queue-attributes --queue-url $QUEUE_URL --attribute-names ApproximateNumberOfMessages` --> should increment by 1 per subscribed queue.

### 4. Build channel-specific processors

Each processor pulls from its queue, checks user preferences, renders the message, and calls the external provider. Implement exponential backoff for retries. [src3]

```python
# Push notification processor (FCM HTTP v1)
import firebase_admin
from firebase_admin import credentials, messaging

cred = credentials.Certificate('service-account.json')
firebase_admin.initialize_app(cred)

def process_push_notification(event: dict):
    """Send push notification via FCM HTTP v1 API."""
    user_id = event['recipient_id']
    device_tokens = get_device_tokens(user_id)  # from device registry

    if not device_tokens:
        return {'status': 'skipped', 'reason': 'no_device_tokens'}

    message = messaging.MulticastMessage(
        tokens=device_tokens[:500],  # FCM limit: 500 per multicast
        notification=messaging.Notification(
            title=event['data'].get('title', 'Notification'),
            body=event['data'].get('body', ''),
        ),
        data={k: str(v) for k, v in event['data'].items()},
        android=messaging.AndroidConfig(priority='high'),
        apns=messaging.APNSConfig(
            headers={'apns-priority': '10'},
            payload=messaging.APNSPayload(
                aps=messaging.Aps(sound='default')
            )
        )
    )
    response = messaging.send_each_for_multicast(message)
    # Prune invalid tokens
    for i, send_response in enumerate(response.responses):
        if send_response.exception and 'UNREGISTERED' in str(send_response.exception):
            remove_device_token(device_tokens[i])
    return {'success': response.success_count, 'failure': response.failure_count}
```

**Verify**: Check FCM response -- `response.success_count` should be > 0 for valid tokens.

### 5. Add in-app notification delivery via WebSocket

For real-time in-app notifications, maintain persistent connections and use Redis Pub/Sub for cross-instance routing. [src4]

```javascript
// Node.js WebSocket in-app notification service
const { Server } = require('socket.io');         // socket.io@4.7.x
const { createClient } = require('redis');        // redis@4.6.x

const io = new Server(3001, { cors: { origin: '*' } });
const redisSub = createClient({ url: process.env.REDIS_URL });
const redisPub = redisSub.duplicate();

async function init() {
  await redisSub.connect();
  await redisPub.connect();

  // Map userId -> Set of socket IDs
  const userSockets = new Map();

  io.on('connection', (socket) => {
    const userId = socket.handshake.auth.userId;
    if (!userSockets.has(userId)) userSockets.set(userId, new Set());
    userSockets.get(userId).add(socket.id);

    socket.on('disconnect', () => {
      userSockets.get(userId)?.delete(socket.id);
      if (userSockets.get(userId)?.size === 0) userSockets.delete(userId);
    });

    socket.on('mark_read', async (notifId) => {
      await markNotificationRead(notifId, userId); // DB update
    });
  });

  // Subscribe to Redis channel for cross-instance delivery
  await redisSub.subscribe('notifications:in_app', (message) => {
    const event = JSON.parse(message);
    const sockets = userSockets.get(event.recipient_id);
    if (sockets) {
      sockets.forEach(socketId => {
        io.to(socketId).emit('notification', event.data);
      });
    }
  });
}

// Called by in-app processor worker
async function deliverInApp(event) {
  // Persist to notification store first
  await saveNotification(event);
  // Publish to Redis for real-time delivery to connected clients
  await redisPub.publish('notifications:in_app', JSON.stringify(event));
}

init().catch(console.error);
```

**Verify**: Connect a test WebSocket client and publish a message to Redis channel `notifications:in_app` --> client should receive the event within 100ms.

### 6. Implement retry logic and dead letter queues

Configure exponential backoff with jitter. After max retries, move to DLQ for manual inspection. [src3]

```python
import time
import random

MAX_RETRIES = 5
BASE_DELAY = 1  # seconds

def process_with_retry(event: dict, processor_fn):
    """Process notification with exponential backoff + jitter."""
    for attempt in range(MAX_RETRIES):
        try:
            return processor_fn(event)
        except TransientError as e:
            if attempt == MAX_RETRIES - 1:
                send_to_dlq(event, str(e))
                raise
            delay = BASE_DELAY * (2 ** attempt) + random.uniform(0, 1)
            time.sleep(delay)
        except PermanentError as e:
            # Invalid token, bad payload -- don't retry
            send_to_dlq(event, str(e))
            raise
```

**Verify**: Check DLQ depth after intentionally sending a malformed payload: `aws sqs get-queue-attributes --queue-url $DLQ_URL --attribute-names ApproximateNumberOfMessages` --> should show 1.

### 7. Add monitoring and alerting

Track delivery success rates per channel, latency percentiles, and queue depths. Alert when delivery rate drops below threshold. [src3]

```yaml
# Prometheus alerting rules (prometheus-rules.yml)
groups:
  - name: notification_alerts
    rules:
      - alert: NotificationDeliveryRateLow
        expr: >
          rate(notifications_delivered_total{status="success"}[5m])
          / rate(notifications_dispatched_total[5m]) < 0.95
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Notification delivery rate below 95% for {{ $labels.channel }}"
      - alert: DLQDepthHigh
        expr: notification_dlq_depth > 100
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Dead letter queue depth exceeds 100 for {{ $labels.channel }}"
```

**Verify**: Trigger a test alert by temporarily lowering the threshold. Check Grafana dashboard shows the alert firing.

## Code Examples
<!-- Keep inline examples <=15 lines. For longer scripts, extract to scripts/ subdirectory
     and link: "Full script: [name.ext](scripts/name.ext) (N lines)" -->

### Python: Send Push Notification via FCM HTTP v1

```python
# Input:  device_token (str), title (str), body (str), data (dict)
# Output: message_id (str) on success

import firebase_admin
from firebase_admin import credentials, messaging  # firebase-admin>=6.4.0

cred = credentials.Certificate('service-account.json')
firebase_admin.initialize_app(cred)

def send_push(token: str, title: str, body: str, data: dict = None) -> str:
    """Send a single push notification via FCM HTTP v1 API."""
    message = messaging.Message(
        token=token,
        notification=messaging.Notification(title=title, body=body),
        data={k: str(v) for k, v in (data or {}).items()},
        android=messaging.AndroidConfig(priority='high'),
        apns=messaging.APNSConfig(
            headers={'apns-priority': '10'},
            payload=messaging.APNSPayload(aps=messaging.Aps(sound='default'))
        )
    )
    return messaging.send(message)  # Returns message ID string
```

### Node.js: In-App Notification Service with Socket.io

```javascript
// Input:  userId (string), notification payload (object)
// Output: Emits 'notification' event to all connected sockets for the user

const { Server } = require('socket.io');    // socket.io@4.7.x
const { createClient } = require('redis');  // redis@4.6.x

const io = new Server(3001, { cors: { origin: '*' } });
const redis = createClient({ url: process.env.REDIS_URL });

const userSockets = new Map(); // userId -> Set<socketId>

io.on('connection', (socket) => {
  const uid = socket.handshake.auth.userId;
  if (!userSockets.has(uid)) userSockets.set(uid, new Set());
  userSockets.get(uid).add(socket.id);
  socket.on('disconnect', () => {
    userSockets.get(uid)?.delete(socket.id);
    if (!userSockets.get(uid)?.size) userSockets.delete(uid);
  });
});

async function notifyUser(userId, payload) {
  const sockets = userSockets.get(userId);
  if (sockets) sockets.forEach(id => io.to(id).emit('notification', payload));
  await redis.publish('notif:in_app', JSON.stringify({ userId, payload }));
}
```

### Python: Email Notification via SendGrid

```python
# Input:  to_email (str), subject (str), html_content (str)
# Output: HTTP status code (202 on success)

from sendgrid import SendGridAPIClient          # sendgrid>=6.11.0
from sendgrid.helpers.mail import Mail, Email, Content

def send_email(to_email: str, subject: str, html_content: str) -> int:
    """Send transactional email via SendGrid."""
    sg = SendGridAPIClient(api_key='SG.your_api_key')
    mail = Mail(
        from_email=Email('noreply@example.com', 'My App'),
        subject=subject,
        to_emails=to_email,
        html_content=Content('text/html', html_content)
    )
    mail.add_header('List-Unsubscribe', '<https://example.com/unsubscribe>')
    response = sg.send(mail)
    return response.status_code  # 202 = accepted
```

## Anti-Patterns

### Wrong: Synchronous notification delivery in the request path

```python
# BAD -- blocks the API response until all notifications are sent
@app.post('/api/orders/{order_id}/ship')
def ship_order(order_id: str):
    order = update_order_status(order_id, 'shipped')
    send_push_notification(order.user_id, 'Your order shipped!')   # blocks 200-500ms
    send_email(order.user_email, 'Order Shipped', render_email())  # blocks 1-3s
    send_sms(order.user_phone, 'Your order shipped!')              # blocks 500ms-2s
    return {'status': 'shipped'}  # user waits 2-5 seconds total
```

### Correct: Asynchronous fan-out via message queue

```python
# GOOD -- publishes event and returns immediately; processors handle delivery async
@app.post('/api/orders/{order_id}/ship')
def ship_order(order_id: str):
    order = update_order_status(order_id, 'shipped')
    publish_notification_event({
        'event_type': 'order.shipped',
        'recipient_id': order.user_id,
        'channels': ['push', 'email', 'sms'],
        'priority': 'high',
        'data': {'order_id': order_id},
        'idempotency_key': f'order_shipped_{order_id}'
    })
    return {'status': 'shipped'}  # returns in <50ms
```

### Wrong: Single shared queue for all channels

```python
# BAD -- email provider outage blocks push and SMS delivery
notifications_queue = Queue('all_notifications')  # single queue

def process_all(event):
    if event['channel'] == 'email':
        send_email(event)       # if SendGrid is down, entire queue backs up
    elif event['channel'] == 'push':
        send_push(event)
    elif event['channel'] == 'sms':
        send_sms(event)
```

### Correct: Separate queues per channel with independent consumers

```python
# GOOD -- channel isolation; email outage does not affect push/SMS
push_queue = Queue('notifications:push')
email_queue = Queue('notifications:email')
sms_queue = Queue('notifications:sms')
inapp_queue = Queue('notifications:in_app')

# Each consumer scales independently based on its queue depth
@push_queue.consumer
def process_push(event):
    send_push(event)

@email_queue.consumer
def process_email(event):
    send_email(event)
```

### Wrong: No idempotency protection on notification delivery

```python
# BAD -- if the worker crashes after sending but before acknowledging,
# the message is redelivered and the user gets a duplicate notification
def process_notification(event):
    send_push(event['token'], event['message'])
    queue.ack(event['receipt_handle'])  # crash here = duplicate send
```

### Correct: Idempotency check before sending

```python
# GOOD -- check if notification was already sent using idempotency key
def process_notification(event):
    idem_key = event['idempotency_key']
    if redis.setnx(f'notif:sent:{idem_key}', '1'):
        redis.expire(f'notif:sent:{idem_key}', 86400)  # TTL: 24h
        send_push(event['token'], event['message'])
    else:
        log.info(f'Duplicate notification skipped: {idem_key}')
    queue.ack(event['receipt_handle'])
```

### Wrong: Ignoring device token lifecycle

```python
# BAD -- using stale tokens wastes API calls and triggers provider throttling
def send_to_all_devices(user_id, message):
    tokens = db.query('SELECT token FROM devices WHERE user_id = ?', user_id)
    for token in tokens:
        fcm.send(token, message)  # never checks if token is still valid
```

### Correct: Prune invalid tokens on provider feedback

```python
# GOOD -- remove tokens that FCM/APNs report as invalid
def send_to_all_devices(user_id, message):
    tokens = db.query('SELECT token FROM devices WHERE user_id = ?', user_id)
    response = fcm.send_multicast(tokens, message)
    for i, result in enumerate(response.responses):
        if result.exception:
            error_code = result.exception.code
            if error_code in ('UNREGISTERED', 'INVALID_ARGUMENT'):
                db.execute('DELETE FROM devices WHERE token = ?', tokens[i])
```

## Common Pitfalls

- **Notification storms during incidents**: A cascading failure triggers millions of alerts simultaneously, overwhelming queues and providers. Fix: implement circuit breakers and per-event-type rate limits (e.g., `max 1 alert per user per event type per 5 minutes`). [src3]
- **Stale device tokens causing FCM/APNs errors**: Users uninstall the app but tokens remain in your database. FCM returns `messaging/registration-token-not-registered`. Fix: prune tokens when providers return 404/410/UNREGISTERED errors; run a weekly token validation batch job. [src1]
- **Email landing in spam due to missing authentication**: Transactional emails without SPF, DKIM, and DMARC records get flagged. Fix: configure DNS records for your sending domain; use a subdomain (e.g., `notifications.example.com`) separate from marketing. [src6]
- **No quiet hours enforcement**: Sending push notifications at 3 AM annoys users and increases opt-out rates. Fix: store user timezone in preferences; defer non-critical notifications to the next active window. [src4]
- **Queue poisoning from malformed payloads**: A single bad message causes repeated processor crashes and redelivery loops. Fix: validate payloads before enqueueing; set `maxReceiveCount` on SQS (or equivalent) to move poison messages to DLQ after 3-5 retries. [src3]
- **Missing idempotency causing duplicate notifications**: At-least-once delivery semantics mean workers may process the same message twice. Fix: include an idempotency key in every notification event; check Redis/DB before sending. [src4]
- **Ignoring provider rate limits**: FCM allows 600K requests/minute for Android; APNs will throttle and drop if you burst too fast. Fix: implement token-bucket rate limiting per provider; use batch APIs (FCM multicast: 500 tokens/call). [src1]
- **Monolithic notification table**: Storing all notification history in a single table degrades query performance after millions of rows. Fix: partition by `created_at` month; archive notifications older than 90 days to cold storage (S3/GCS). [src3]

## Diagnostic Commands

```bash
# Check SQS queue depth (AWS)
aws sqs get-queue-attributes --queue-url $QUEUE_URL \
  --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible

# Check DLQ depth for failed notifications
aws sqs get-queue-attributes --queue-url $DLQ_URL \
  --attribute-names ApproximateNumberOfMessages

# Verify FCM service account is valid
firebase projects:list  # Firebase CLI -- should list your project

# Test push notification delivery (FCM)
curl -X POST "https://fcm.googleapis.com/v1/projects/YOUR_PROJECT/messages:send" \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  -d '{"message":{"token":"DEVICE_TOKEN","notification":{"title":"Test","body":"Hello"}}}'

# Check Redis Pub/Sub channel subscribers (in-app delivery)
redis-cli PUBSUB NUMSUB notifications:in_app

# Monitor email delivery via SendGrid
curl -s "https://api.sendgrid.com/v3/stats?start_date=2026-02-23" \
  -H "Authorization: Bearer $SENDGRID_API_KEY" | jq '.[] | {delivered: .stats[0].metrics.delivered}'

# Check Kafka consumer lag (notification processor)
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group notification-push-processor
```

## Version History & Compatibility

| Version / Service | Status | Breaking Changes | Migration Notes |
|---|---|---|---|
| FCM HTTP v1 API | Current (2024+) | Legacy HTTP/XMPP removed June 2024 | Migrate to `https://fcm.googleapis.com/v1/projects/{project}/messages:send` with OAuth2 |
| APNs HTTP/2 | Current (2016+) | Binary provider protocol removed 2021 | Use HTTP/2 on port 443; prefer token-based auth over certificate-based |
| SendGrid v3 API | Current | v2 API deprecated | Use `/v3/mail/send` endpoint; API keys replace username/password |
| Socket.io 4.x | Current | v2->v4: namespace middleware changed | Upgrade client and server together; test reconnection logic |
| AWS SNS/SQS | Stable | FIFO queues added 2020 | Use FIFO for ordered delivery; standard queues for throughput |

## When to Use / When Not to Use

| Use When | Don't Use When | Use Instead |
|---|---|---|
| Multi-channel delivery (push + email + SMS + in-app) is required | Only need simple webhook callbacks to a single endpoint | Webhook delivery system with retry |
| Scale exceeds 10K notifications/day and growing | Fewer than 100 notifications/day with a single channel | Direct API calls to FCM/SendGrid from your app |
| Need user preference management and quiet hours | Sending only system-to-system alerts (no human recipients) | Event bus (Kafka/RabbitMQ) without notification layer |
| Regulatory compliance (CAN-SPAM, GDPR) matters | Internal dev notifications only (Slack/Discord bots) | Slack Incoming Webhooks or Discord Bot API |
| Need delivery tracking, analytics, and audit trails | Fire-and-forget logging with no delivery guarantees needed | Simple log aggregation (ELK/Loki) |

## Important Caveats

- FCM delivery is best-effort on Android when the app is force-stopped; only high-priority data messages wake the device, and even then OEM battery optimizations (Xiaomi, Samsung, Huawei) may block delivery
- APNs device tokens change when a user restores from backup to a new device; always re-register tokens on app launch
- Email delivery reputation is per-IP and per-domain; new IPs must be "warmed up" gradually (start with 50-100 emails/day, increase over 4-6 weeks) or deliverability suffers
- WebSocket connections are stateful; horizontal scaling requires a shared Pub/Sub layer (Redis, NATS) to route messages to the correct server instance
- SMS costs vary dramatically by country ($0.01-$0.10+ per message); implement per-country rate limits and cost caps to prevent bill shock
- At-least-once delivery is the practical guarantee for distributed notification systems; design clients to handle duplicate notifications gracefully (idempotent UI updates)

## Related Units
<!-- Generated from related_kos frontmatter -->

- [Message Queue Architecture (Kafka vs RabbitMQ vs SQS)](/software/system-design/message-queues/2026)
- [Rate Limiting and Throttling Patterns](/software/system-design/rate-limiting/2026)
- [Event-Driven Architecture](/software/system-design/event-driven-architecture/2026)
- [Debugging Notification Delivery Failures](/software/debugging/notification-delivery-failures/2026)
- [Webhook Delivery System Design](/software/system-design/webhook-system/2026)
- [Real-Time Chat System Design](/software/system-design/chat-system/2026)