--- # === IDENTITY === id: software/system-design/multi-layer-caching/2026 canonical_question: "How do I design a multi-layer caching strategy?" aliases: - "How to implement multi-tier caching architecture?" - "Multi-level cache design for distributed systems" - "L1 L2 caching strategy system design" - "How to combine CDN, Redis, and application caching?" entity_type: software_reference domain: software > system-design > multi_layer_caching region: global jurisdiction: global temporal_scope: 2020-2026 # === VERIFICATION === last_verified: 2026-02-23 confidence: 0.93 version: 1.0 first_published: 2026-02-23 # === TEMPORAL VALIDITY === temporal_validity: status: stable last_breaking_change: null next_review: 2026-08-22 change_sensitivity: low # === CONSTRAINTS === constraints: - "Every cache layer must have a TTL -- unbounded caches cause stale data bugs and memory exhaustion" - "Cache invalidation must propagate from the source of truth outward to all layers; never rely on TTL expiry alone for consistency-critical data" - "Never cache user-specific or authenticated data in shared layers (CDN, reverse proxy) without Vary headers or cache keys that include auth tokens" - "Cache stampede protection (locking, request coalescing, or probabilistic early expiration) is mandatory for any key with >100 req/s" - "Monitor cache hit ratios per layer -- a layer with <70% hit rate is likely misconfigured or caching the wrong data" # === SKIP CONDITIONS === skip_this_unit_if: - condition: "Need to design a CDN architecture specifically" use_instead: "software/system-design/cdn-design/2026" - condition: "Debugging Redis memory issues specifically" use_instead: "software/debugging/redis-memory-issues/2026" - condition: "Need CPU-level L1/L2/L3 hardware cache optimization" use_instead: "CPU cache hierarchy is a hardware topic, not covered here" # === AGENT HINTS === inputs_needed: - key: "traffic_volume" question: "What is the expected read QPS (queries per second)?" type: choice options: ["<1K QPS (small)", "1K-100K QPS (medium)", ">100K QPS (large)"] - key: "consistency_requirement" question: "How important is data freshness vs latency?" type: choice options: ["strong consistency (seconds)", "eventual consistency (minutes)", "best-effort (hours)"] - key: "data_type" question: "What type of data are you caching?" type: choice options: ["static assets (images, CSS, JS)", "API responses (JSON)", "database query results", "computed/aggregated data", "session data"] # === DISTRIBUTION === canonical_source: "https://knowledgelib.io/software/system-design/multi-layer-caching/2026" suggested_citation: "Source: knowledgelib.io -- AI Knowledge Library (verified 2026-02-23)" # === RELATED UNITS === related_kos: depends_on: [] related_to: - id: "software/system-design/url-shortener/2026" label: "URL Shortener System Design" - id: "software/debugging/redis-memory-issues/2026" label: "Redis Memory Issues" solves: [] alternative_to: [] often_confused_with: - id: "software/system-design/cdn-design/2026" label: "CDN Architecture Design" # === SOURCES (7 authoritative sources) === sources: - id: src1 title: "Database Caching Strategies Using Redis -- Caching Patterns" author: AWS url: https://docs.aws.amazon.com/whitepapers/latest/database-caching-strategies-using-redis/caching-patterns.html type: official_docs published: 2024-03-15 reliability: high - id: src2 title: "TAO: Facebook's Distributed Data Store for the Social Graph" author: Bronson et al. (Meta/USENIX ATC 2013) url: https://www.usenix.org/system/files/conference/atc13/atc13-bronson.pdf type: academic_paper published: 2013-06-25 reliability: authoritative - id: src3 title: "Caching for System Design Interviews" author: Hello Interview url: https://www.hellointerview.com/learn/system-design/core-concepts/caching type: technical_blog published: 2024-11-01 reliability: high - id: src4 title: "Cloudflare Tiered Cache Documentation" author: Cloudflare url: https://developers.cloudflare.com/cache/how-to/tiered-cache/ type: official_docs published: 2025-01-10 reliability: high - id: src5 title: "Cache Optimization: Strategies to Cut Latency and Cloud Cost" author: Redis url: https://redis.io/blog/guide-to-cache-optimization-strategies/ type: technical_blog published: 2025-03-20 reliability: high - id: src6 title: "Thundering Herd Problem: Preventing the Stampede" author: Distributed Computing Musings url: https://distributed-computing-musings.com/2025/08/thundering-herd-problem-preventing-the-stampede/ type: technical_blog published: 2025-08-15 reliability: moderate_high - id: src7 title: "Design a Distributed Cache System (Step-by-Step Guide)" author: System Design Handbook url: https://www.systemdesignhandbook.com/guides/design-a-distributed-cache-system/ type: technical_blog published: 2024-09-20 reliability: high --- # Multi-Layer Caching Strategy Design ## TL;DR - **Bottom line**: A multi-layer caching strategy places caches at 5 tiers -- browser, CDN/edge, reverse proxy/API gateway, application (in-process), and distributed cache (Redis/Memcached) -- each absorbing traffic before it reaches the database, reducing latency from ~50ms to <5ms for hot data and cutting origin load by 90-99%. - **Key tool/command**: `Cache-Control: public, s-maxage=3600, stale-while-revalidate=60` for CDN; `redis.setex(key, TTL, value)` for distributed cache - **Watch out for**: Cache stampede -- when a hot key expires and thousands of concurrent requests simultaneously hit the database. Use request coalescing or probabilistic early expiration. - **Works with**: Any language/stack. Core components: CDN (Cloudflare, CloudFront), Redis/Memcached, in-process cache (Guava, node-cache, lru-cache), any relational or NoSQL database. ## Constraints - Every cache layer must have a TTL -- unbounded caches cause stale data bugs and memory exhaustion - Cache invalidation must propagate from the source of truth outward to all layers; never rely on TTL expiry alone for consistency-critical data - Never cache user-specific or authenticated data in shared layers (CDN, reverse proxy) without `Vary` headers or cache keys that include authentication context - Cache stampede protection (distributed locking, request coalescing, or probabilistic early expiration) is mandatory for any key with >100 req/s [src6] - Monitor cache hit ratios per layer -- a layer with <70% hit rate is likely misconfigured or caching the wrong data [src5] ## Quick Reference | Layer | Role | Technology Options | Typical TTL | Latency | Scaling Strategy | |---|---|---|---|---|---| | Browser Cache | Stores static assets and API responses locally on user device | HTTP Cache-Control headers, Service Workers | 1 hour - 1 year (immutable for hashed assets) | 0ms | Per-client; no server cost | | CDN / Edge Cache | Serves cached content from geographically nearest edge node | Cloudflare, CloudFront, Fastly, Akamai | 5 min - 24 hours | 1-20ms | Automatic global distribution; tiered caching for origin shielding [src4] | | Reverse Proxy / API Gateway | Caches API responses at the gateway level before reaching app servers | Nginx, Varnish, Kong, AWS API Gateway | 30s - 5 min | 1-5ms | Horizontal scaling; shared cache across app instances | | Application Cache (L1) | In-process memory cache; fastest but per-instance (not shared) | Guava (Java), lru-cache (Node), cachetools (Python) | 30s - 5 min | <0.1ms | Per-instance; scales with app replicas; risk of inconsistency | | Distributed Cache (L2) | Shared cache across all app instances; single source of cached truth | Redis Cluster, Memcached, KeyDB | 5 min - 1 hour | 1-5ms (network hop) | Hash-based sharding, read replicas; 80-20 rule applies [src1] | | Database Query Cache | Built-in query result caching at the database level | MySQL query cache (deprecated), PostgreSQL pg_stat_statements, materialized views | Varies | 5-20ms | Limited; use only for expensive aggregations | | Write-Behind Buffer | Absorbs writes and flushes to DB asynchronously | Redis + background worker, Kafka | N/A (write path) | <1ms write acknowledgment | Decouples write latency from DB; risk of data loss on crash | | Session Cache | Stores user session data for stateless app servers | Redis, Memcached, DynamoDB | 30 min - 24 hours | 1-5ms | Partition by user ID; sticky sessions as fallback | ## Decision Tree > Full script: [decision-tree.txt](scripts/decision-tree.txt) (32 lines) ``` START |-- What type of data? | |-- Static assets (images, CSS, JS, fonts)? | | |-- Use Browser Cache (Cache-Control: immutable) + CDN | | +-- Set long TTL (1 year) with content-hash in filename for busting # ... (see full script) ``` ## Step-by-Step Guide ### 1. Identify cache-worthy data and access patterns Analyze your read-to-write ratio, data staleness tolerance, and hotspot distribution. The 80-20 rule applies: 20% of keys typically serve 80% of traffic. [src1] ``` Data Analysis Checklist: - Read-to-write ratio per entity type (>10:1 is cache-friendly) - p50/p95/p99 latency of uncached DB queries - Hotspot analysis: top 1000 keys by request frequency - Staleness tolerance: seconds, minutes, or hours? - Data size per cached object (affects memory budget) Memory Budget Formula: cache_size = hot_key_count * avg_object_size * 1.3 (overhead) Example: 100K hot keys * 2KB avg * 1.3 = ~260MB Redis ``` **Verify**: Run `SELECT query, calls, mean_exec_time FROM pg_stat_statements ORDER BY calls DESC LIMIT 20` to find your most frequent queries. ### 2. Implement L1 in-process application cache Add a local in-memory cache with short TTL to each application instance. This eliminates network round-trips for the hottest data. [src3] ```python # Python: using cachetools (pip install cachetools==5.3.2) from cachetools import TTLCache import threading # L1 cache: 1000 items max, 60-second TTL l1_cache = TTLCache(maxsize=1000, ttl=60) l1_lock = threading.Lock() def get_from_l1(key: str): """Check L1 (in-process) cache first.""" with l1_lock: return l1_cache.get(key) def set_in_l1(key: str, value): """Store in L1 cache.""" with l1_lock: l1_cache[key] = value ``` **Verify**: Monitor L1 hit rate. If <50%, increase maxsize or TTL. If >95% on all keys, your data may be static enough for CDN-only caching. ### 3. Set up L2 distributed cache with Redis Deploy Redis as the shared cache layer across all application instances. Use the cache-aside pattern: check cache first, fall back to DB on miss, then populate cache. [src1] ```python # Python: Redis cache-aside pattern (pip install redis==5.0.1) import redis import json r = redis.Redis(host='redis-primary', port=6379, decode_responses=True) L2_TTL = 300 # 5 minutes def get_with_caching(key: str, db_fetch_fn): """Two-layer lookup: L1 -> L2 (Redis) -> Database.""" # 1. Check L1 (in-process) value = get_from_l1(key) if value is not None: return value # L1 hit (~0.1ms) # 2. Check L2 (Redis) cached = r.get(f"cache:{key}") if cached: value = json.loads(cached) set_in_l1(key, value) # Promote to L1 return value # L2 hit (~1-5ms) # 3. Cache miss -> query database value = db_fetch_fn(key) # DB query (~10-100ms) if value is not None: r.setex(f"cache:{key}", L2_TTL, json.dumps(value)) set_in_l1(key, value) return value ``` **Verify**: `redis-cli INFO stats | grep keyspace` -- target >85% hit rate. `redis-cli INFO memory` to check memory usage vs maxmemory. ### 4. Configure CDN caching with proper headers Set Cache-Control headers to leverage CDN edge caching for public data. Use `s-maxage` for CDN TTL independent of browser TTL. [src4] ```python # Python (FastAPI): setting cache headers for CDN from fastapi import FastAPI from fastapi.responses import JSONResponse app = FastAPI() @app.get("/api/products/{product_id}") async def get_product(product_id: str): product = get_with_caching(f"product:{product_id}", fetch_product_from_db) return JSONResponse( content=product, headers={ # Browser caches for 60s, CDN caches for 5min "Cache-Control": "public, max-age=60, s-maxage=300", # CDN serves stale content for 60s while revalidating "CDN-Cache-Control": "public, s-maxage=300, stale-while-revalidate=60", # Vary ensures different cache entries per encoding "Vary": "Accept-Encoding", } ) ``` **Verify**: `curl -I https://yourcdn.com/api/products/123` -- check for `cf-cache-status: HIT` (Cloudflare) or `x-cache: Hit from cloudfront` (AWS). ### 5. Implement cache invalidation strategy Choose between TTL-based expiry, event-driven invalidation, or a hybrid approach depending on consistency requirements. [src1] ```python # Event-driven invalidation: publish cache-bust events on writes import redis r = redis.Redis(host='redis-primary', port=6379, decode_responses=True) def update_product(product_id: str, new_data: dict): """Write-through: update DB, then invalidate all cache layers.""" # 1. Write to database (source of truth) db.update("products", product_id, new_data) # 2. Invalidate L2 (Redis) r.delete(f"cache:product:{product_id}") # 3. Publish invalidation event for L1 caches on other instances r.publish("cache:invalidate", f"product:{product_id}") # 4. Purge CDN cache (Cloudflare example) # requests.post(f"https://api.cloudflare.com/client/v4/zones/{zone_id}/purge_cache", # headers={"Authorization": f"Bearer {cf_token}"}, # json={"files": [f"https://example.com/api/products/{product_id}"]}) # Subscribe to invalidation events (run in each app instance) def cache_invalidation_listener(): pubsub = r.pubsub() pubsub.subscribe("cache:invalidate") for message in pubsub.listen(): if message["type"] == "message": key = message["data"] # Clear from L1 with l1_lock: l1_cache.pop(key, None) ``` **Verify**: After an update, check that subsequent reads return the new value within your expected consistency window. ### 6. Add cache stampede protection Implement request coalescing or distributed locking to prevent thundering herd when hot keys expire. [src6] ```python # Distributed lock to prevent stampede on cache miss import redis import time import json r = redis.Redis(host='redis-primary', port=6379, decode_responses=True) LOCK_TTL = 5 # seconds def get_with_stampede_protection(key: str, db_fetch_fn, cache_ttl=300): """Cache-aside with distributed lock to prevent stampede.""" # 1. Try cache first cached = r.get(f"cache:{key}") if cached: return json.loads(cached) # 2. Try to acquire lock lock_key = f"lock:{key}" acquired = r.set(lock_key, "1", nx=True, ex=LOCK_TTL) if acquired: try: # 3. This instance rebuilds the cache value = db_fetch_fn(key) if value is not None: r.setex(f"cache:{key}", cache_ttl, json.dumps(value)) return value finally: r.delete(lock_key) else: # 4. Another instance is rebuilding -- wait and retry for _ in range(50): # 50 * 100ms = 5s max wait time.sleep(0.1) cached = r.get(f"cache:{key}") if cached: return json.loads(cached) # Fallback: hit DB directly if lock holder is slow return db_fetch_fn(key) ``` **Verify**: Under load testing with a single hot key, only 1 request should reach the database per cache miss event. ## Code Examples ### Python: Complete Multi-Layer Cache Client > Full script: [python-complete-multi-layer-cache-client.py](scripts/python-complete-multi-layer-cache-client.py) (48 lines) ```python # Input: A cache key and a callable that fetches from the database # Output: Cached value from fastest available layer import redis import json from cachetools import TTLCache # ... (see full script) ``` ### HTTP: CDN Cache-Control Headers Reference ```http # Static immutable assets (hashed filenames like app.a1b2c3.js) Cache-Control: public, max-age=31536000, immutable # Public API responses (CDN + browser caching) Cache-Control: public, max-age=60, s-maxage=300 CDN-Cache-Control: public, s-maxage=300, stale-while-revalidate=60 # Private user-specific data (no CDN, browser only) Cache-Control: private, max-age=0, no-store # Stale-while-revalidate pattern for best UX Cache-Control: public, max-age=300, stale-while-revalidate=60, stale-if-error=86400 # Vary header to prevent serving wrong cached response Vary: Accept-Encoding, Authorization ``` ## Anti-Patterns ### Wrong: Single global TTL for all cache entries ```python # BAD -- one TTL for everything means stale user profiles or wasted cache on static data GLOBAL_TTL = 3600 cache.setex(f"user:{uid}", GLOBAL_TTL, data) # User data stale for 1 hour cache.setex(f"config:{key}", GLOBAL_TTL, config) # Static config only cached 1 hour cache.setex(f"feed:{uid}", GLOBAL_TTL, feed) # Social feed stale for 1 hour ``` ### Correct: TTL per data type based on staleness tolerance ```python # GOOD -- TTL matches data volatility TTL_CONFIG = {"user_profile": 300, "static_config": 86400, "social_feed": 30, "product_catalog": 3600} cache.setex(f"user:{uid}", TTL_CONFIG["user_profile"], data) # 5 min cache.setex(f"config:{key}", TTL_CONFIG["static_config"], config) # 24 hours cache.setex(f"feed:{uid}", TTL_CONFIG["social_feed"], feed) # 30 seconds ``` ### Wrong: No cache stampede protection on hot keys ```python # BAD -- when a hot key expires, all 10,000 concurrent requests hit the DB [src6] def get_product(product_id): cached = redis.get(f"product:{product_id}") if cached: return json.loads(cached) # 10,000 requests all reach here simultaneously product = db.query("SELECT * FROM products WHERE id = %s", [product_id]) redis.setex(f"product:{product_id}", 300, json.dumps(product)) return product ``` ### Correct: Request coalescing with distributed lock ```python # GOOD -- only one request rebuilds the cache; others wait or get stale data [src6] def get_product(product_id): cached = redis.get(f"product:{product_id}") if cached: return json.loads(cached) lock = redis.set(f"lock:product:{product_id}", "1", nx=True, ex=5) if lock: try: product = db.query("SELECT * FROM products WHERE id = %s", [product_id]) redis.setex(f"product:{product_id}", 300, json.dumps(product)) return product finally: redis.delete(f"lock:product:{product_id}") else: time.sleep(0.1) # Brief wait, then retry cache return json.loads(redis.get(f"product:{product_id}") or "null") ``` ### Wrong: Caching authenticated data in CDN without Vary ```python # BAD -- CDN serves User A's dashboard to User B @app.get("/api/dashboard") async def dashboard(user: User): data = get_user_dashboard(user.id) return JSONResponse(content=data, headers={ "Cache-Control": "public, s-maxage=300" # CDN caches for ALL users! }) ``` ### Correct: Private cache or per-user cache keys ```python # GOOD -- user-specific data never enters shared CDN cache @app.get("/api/dashboard") async def dashboard(user: User): data = get_user_dashboard(user.id) return JSONResponse(content=data, headers={ "Cache-Control": "private, max-age=60" # Browser-only, per-user }) # For CDN-cacheable user data, use Vary or surrogate keys: # Vary: Authorization (creates separate cache entry per auth token) ``` ### Wrong: Invalidating only L2 and forgetting L1 ```python # BAD -- L1 in-process caches on other instances still serve stale data def update_product(product_id, new_data): db.update("products", product_id, new_data) redis.delete(f"product:{product_id}") # L2 cleared # But L1 caches on 10 other app instances still have old data for up to 60s! ``` ### Correct: Broadcast invalidation across all layers ```python # GOOD -- pub/sub ensures all L1 caches are cleared [src2] def update_product(product_id, new_data): db.update("products", product_id, new_data) redis.delete(f"product:{product_id}") # Clear L2 redis.publish("cache:invalidate", f"product:{product_id}") # Notify all L1s ``` ## Common Pitfalls - **Cache stampede / thundering herd**: When a popular cache key expires, thousands of concurrent requests simultaneously query the database. Fix: implement distributed locking (`SET key NX EX 5`), request coalescing, or probabilistic early expiration (`currentTime > expiry - TTL * beta * log(random())`). [src6] - **Stale L1 after L2 invalidation**: In-process L1 caches on other instances are not aware when L2 is invalidated. Fix: use Redis Pub/Sub or a message bus to broadcast invalidation events to all instances; keep L1 TTL short (30-60s). [src2] - **Unbounded cache growth**: Caches without maxsize or maxmemory fill RAM and cause OOM. Fix: always set `maxmemory` in Redis config and `maxsize` in L1 caches. Use LRU or LFU eviction policies. [src5] - **Cache-DB inconsistency on write**: Writing to DB then cache (or vice versa) without atomicity causes races. Fix: use write-through pattern (update cache on successful DB write) or invalidate-on-write (delete cache key, let next read repopulate). Never update cache first. [src1] - **CDN caching authenticated responses**: Setting `Cache-Control: public` on user-specific endpoints causes data leaks. Fix: use `private` or `no-store` for authenticated responses; use `Vary: Authorization` only if you understand the CDN's behavior with that header. [src4] - **No monitoring of per-layer hit rates**: Without visibility, a misconfigured layer silently passes all traffic through. Fix: instrument each cache layer with hit/miss counters; alert when hit rate drops below 70%. [src5] - **Over-caching rarely accessed data**: Caching every DB query wastes memory. The 80-20 rule means 80% of cache memory is wasted on cold data. Fix: only cache data accessed >N times per minute; use Redis `OBJECT FREQ` with LFU eviction to verify. [src1] - **TTL too long for price/inventory data**: E-commerce sites caching product prices for hours show stale prices. Fix: use event-driven invalidation for price changes and short TTL (30-60s) as a safety net. [src3] ## Diagnostic Commands ```bash # Check Redis cache hit ratio redis-cli INFO stats | grep -E "keyspace_hits|keyspace_misses" # Check Redis memory usage and eviction policy redis-cli INFO memory | grep -E "used_memory_human|maxmemory_human|maxmemory_policy" # Monitor Redis commands in real-time (careful in production -- high overhead) redis-cli MONITOR | head -50 # Check slow Redis commands redis-cli SLOWLOG GET 10 # Verify CDN cache status (Cloudflare) curl -sI https://example.com/api/products/123 | grep -i "cf-cache-status" # Verify CDN cache status (CloudFront) curl -sI https://example.com/api/products/123 | grep -i "x-cache" # Check Cache-Control headers on response curl -sI https://example.com/api/products/123 | grep -i "cache-control" # Redis key count and memory per prefix redis-cli --scan --pattern "cache:product:*" | wc -l redis-cli MEMORY USAGE "cache:product:123" # Application-level cache stats (if exposed via metrics endpoint) curl -s http://localhost:8080/metrics | grep cache_hit ``` ## When to Use / When Not to Use | Use When | Don't Use When | Use Instead | |---|---|---| | Read-to-write ratio >10:1 and data can tolerate seconds/minutes of staleness | Data must be real-time consistent (financial transactions, inventory decrements) | Direct DB reads with optimistic locking | | Database query latency >50ms and query is repeated frequently | Each request has unique parameters (e.g., full-text search with user input) | Query optimization, read replicas, or search engines (Elasticsearch) | | Multiple app instances need shared cached state | Single app instance with low traffic (<100 QPS) | In-process cache only (no Redis overhead) | | Static or semi-static content (product catalogs, config, reference data) | Data changes per-request (real-time stock tickers, live scores) | WebSockets or Server-Sent Events with no caching | | Need to reduce cloud costs by offloading DB/API compute | Cache infrastructure cost exceeds DB query cost | Scale the database directly (read replicas, vertical scaling) | ## Important Caveats - Facebook TAO processes 1B+ reads/sec using a multi-layer leader/follower cache architecture. TAO chose eventual consistency with master/slave replication across data centers -- writes always go to the master, but read misses can be served locally. This trade-off is appropriate for social graph data but not for financial systems. [src2] - `stale-while-revalidate` provides the best user experience for most web applications: users always get a fast response (possibly stale), while a background refresh ensures the next request gets fresh data. However, it requires CDN support (Cloudflare, CloudFront, Fastly all support it). [src4] - Redis persistence (RDB/AOF) should be configured on cache nodes if you cannot tolerate a cold start that hammers the database. For pure caches, persistence can be disabled for better performance. [src5] - Cloudflare's Tiered Cache creates an upper-tier/lower-tier hierarchy that reduces origin requests by up to 90%. Enable it before adding more application-level caching complexity. [src4] - In-process L1 caches create consistency challenges proportional to the number of app instances. With 50 instances and 60-second L1 TTL, a cached item may be stale on some instances for up to 60 seconds after invalidation. Accept this or reduce L1 TTL. ## Related Units - [URL Shortener System Design](/software/system-design/url-shortener/2026) - [Redis Memory Issues Debugging](/software/debugging/redis-memory-issues/2026) - [CDN Architecture Design](/software/system-design/cdn-design/2026) - [Rate Limiter System Design](/software/system-design/rate-limiter/2026)