--- # === IDENTITY === id: software/system-design/social-media-feed/2026 canonical_question: "How do I design a social media feed system (Twitter/Instagram clone)?" aliases: - "How to build a Twitter-like news feed at scale" - "Design a scalable social media timeline system" - "Fan-out-on-write vs fan-out-on-read for feeds" - "System design interview: design Instagram feed" entity_type: software_reference domain: software > system-design > social_media_feed region: global jurisdiction: global temporal_scope: 2020-2026 # === VERIFICATION === last_verified: 2026-02-23 confidence: 0.91 version: 1.0 first_published: 2026-02-23 # === TEMPORAL VALIDITY === temporal_validity: status: stable last_breaking_change: null next_review: 2026-08-22 change_sensitivity: medium # === CONSTRAINTS === constraints: - "Fan-out-on-write must be disabled for users with >500K followers to avoid write amplification storms" - "Timeline cache must have a TTL and size cap (e.g. 800 tweets) to bound memory usage per user" - "Never store media blobs in the primary database; use object storage (S3/GCS) with CDN delivery" - "Feed ranking models require A/B testing infrastructure before production deployment" # === SKIP CONDITIONS === skip_this_unit_if: - condition: "Building a simple chronological feed for <10K users" use_instead: "A standard relational query with pagination is sufficient" - condition: "Designing a recommendation-only feed (no follow graph)" use_instead: "Recommendation system design (collaborative filtering / content-based)" # === AGENT HINTS === inputs_needed: - key: "scale" question: "What is your expected daily active user (DAU) count?" type: choice options: ["<10K", "10K-1M", "1M-100M", ">100M"] - key: "feed_type" question: "Do you need a chronological feed, ranked feed, or both?" type: choice options: ["chronological", "ranked", "hybrid"] - key: "media_type" question: "What content types will the feed contain?" type: choice options: ["text-only", "text+images", "text+images+video", "video-first"] # === DISTRIBUTION === canonical_source: "https://knowledgelib.io/software/system-design/social-media-feed/2026" suggested_citation: "Source: knowledgelib.io — AI Knowledge Library (verified 2026-02-23)" # === RELATED UNITS === related_kos: depends_on: [] related_to: - id: "software/system-design/chat-system/2026" label: "Chat System Design" - id: "software/system-design/notification-system/2026" label: "Notification System Design" solves: [] alternative_to: [] often_confused_with: - id: "software/system-design/search-engine/2026" label: "Search Engine Design (retrieval vs feed push)" # === SOURCES (7 authoritative sources) === sources: - id: src1 title: "Designing a Scalable News Feed System (Step-by-Step)" author: Ashish Pratap Singh url: https://blog.algomaster.io/p/designing-a-scalable-news-feed-system type: technical_blog published: 2024-06-15 reliability: high - id: src2 title: "System Design Primer: Design Twitter" author: Donne Martin url: https://github.com/donnemartin/system-design-primer/blob/master/solutions/system_design/twitter/README.md type: community_resource published: 2023-01-01 reliability: high - id: src3 title: "Twitter's Tough Architectural Decision" author: Denny Sam url: https://softwareengineeringwk.substack.com/p/twitter-architecture type: technical_blog published: 2023-08-20 reliability: moderate_high - id: src4 title: "How Instagram Suggests New Content" author: Meta Engineering url: https://engineering.fb.com/2020/12/10/web/how-instagram-suggests-new-content/ type: technical_blog published: 2020-12-10 reliability: authoritative - id: src5 title: "Scaling Instagram Explore Recommendations System" author: Meta Engineering url: https://engineering.fb.com/2023/08/09/ml-applications/scaling-instagram-explore-recommendations-system/ type: technical_blog published: 2023-08-09 reliability: authoritative - id: src6 title: "How to Design Social Media News Feed" author: Design Gurus url: https://www.designgurus.io/blog/design-social-media-news-feed type: community_resource published: 2024-01-15 reliability: moderate_high - id: src7 title: "Social Media Feed System Design - Architecture, Algorithms & Best Practices" author: JavaTechOnline url: https://javatechonline.com/social-media-feed-system-design/ type: technical_blog published: 2024-03-10 reliability: moderate_high --- # Social Media Feed System Design ## TL;DR - **Bottom line**: A social media feed requires a hybrid fan-out architecture -- fan-out-on-write for normal users (precompute timelines into Redis) and fan-out-on-read for celebrity accounts (>500K followers) to avoid write amplification. - **Key tool/command**: `LPUSH user:{id}:timeline {tweet_id}` (Redis list per user for O(1) feed reads) - **Watch out for**: Unbounded fan-out for high-follower accounts -- a single celebrity tweet can generate millions of writes and lag the entire pipeline. - **Works with**: Any stack; core patterns are language-agnostic. Redis + Kafka + PostgreSQL/Cassandra is the most battle-tested combination. ## Constraints - Fan-out-on-write must be disabled for users with >500K followers to avoid write amplification storms - Timeline cache must have a TTL and size cap (e.g. 800 tweets per user) to bound memory usage - Never store media blobs in the primary database; use object storage (S3/GCS) with CDN delivery - Feed ranking models require A/B testing infrastructure before production deployment - Eventual consistency is acceptable for feed delivery (seconds-level lag), but like/reply counts need at-least-once delivery guarantees ## Quick Reference | Component | Role | Technology Options | Scaling Strategy | |---|---|---|---| | API Gateway | Rate limiting, auth, routing | Nginx, Kong, AWS API Gateway | Horizontal scaling behind LB | | Post Service | Create/store posts and metadata | Python/Go + PostgreSQL | Shard by user_id, master-slave replication | | Fan-Out Service | Distribute posts to follower timelines | Go/Java workers + Kafka consumers | Partition Kafka by user_id, scale consumers horizontally | | Timeline Service | Serve precomputed home feeds | Node.js/Go + Redis | Redis Cluster with consistent hashing | | Social Graph Service | Manage follow/unfollow relationships | Go/Java + Graph DB or Redis | Adjacency list in Redis, Neo4j for deep queries | | User Service | Profiles, auth, preferences | Any framework + PostgreSQL | Read replicas, cache hot profiles | | Media Service | Upload, process, serve images/video | Python/Go + S3/GCS + CDN | Pre-signed uploads, CDN edge caching | | Search Service | Full-text search on posts | Elasticsearch, Apache Solr | Sharded index, scatter-gather queries | | Notification Service | Push/email/in-app alerts on engagement | Go/Python + Kafka + FCM/APNs | Async via message queue, batch delivery | | Ranking Service | ML-based feed personalization | Python + TensorFlow/PyTorch | Feature store + model serving (TF Serving, Triton) | | CDN | Static asset and media delivery | CloudFront, Cloudflare, Fastly | Edge PoPs, cache invalidation via purge API | | Message Queue | Async communication between services | Apache Kafka, RabbitMQ, SQS | Partitioned topics, consumer groups | | Cache Layer | Reduce DB load for hot data | Redis, Memcached | Redis Cluster, LRU eviction, ~800 items per timeline | | Object Storage | Persistent media file storage | AWS S3, Google Cloud Storage | Lifecycle policies, cross-region replication | ## Decision Tree ``` START |-- Expected DAU? | |-- <10K users | | --> Simple pull-based feed: query DB at read time, paginate with cursor | | --> Stack: PostgreSQL + application server, no cache needed | |-- 10K - 1M users | | --> Fan-out-on-write for all users | | --> Precompute timelines into Redis on every post | | --> Add Kafka for async fan-out processing | |-- 1M - 100M users | | |-- Any user with >500K followers? | | | |-- YES --> Hybrid approach: | | | | --> Fan-out-on-write for normal users (<500K followers) | | | | --> Fan-out-on-read for celebrities (merge at serve time) | | | |-- NO --> Pure fan-out-on-write with Redis Cluster | | |-- Feed type? | | | |-- Chronological --> Sort by timestamp in Redis (sorted set) | | | |-- Ranked --> Add Ranking Service with ML model | | | |-- Hybrid --> Chronological base + lightweight re-ranking | |-- >100M users | --> Full hybrid fan-out + ML ranking pipeline | --> Multi-region deployment with geo-sharding | --> Edge caching for feed responses | --> Real-time feature store for ranking signals ``` ## Step-by-Step Guide ### 1. Define the data model and API contracts Design the core entities: Users, Posts, Follows, Likes, Comments. Define REST or gRPC endpoints for create-post, get-timeline, follow/unfollow. [src1] ```sql -- Core tables (PostgreSQL) CREATE TABLE users ( user_id BIGINT PRIMARY KEY, username VARCHAR(64) UNIQUE NOT NULL, created_at TIMESTAMPTZ DEFAULT NOW() ); CREATE TABLE posts ( post_id BIGINT PRIMARY KEY, author_id BIGINT NOT NULL REFERENCES users(user_id), content TEXT NOT NULL, media_url TEXT, created_at TIMESTAMPTZ DEFAULT NOW(), like_count INT DEFAULT 0, INDEX idx_author_time (author_id, created_at DESC) ); CREATE TABLE follows ( follower_id BIGINT NOT NULL, followee_id BIGINT NOT NULL, created_at TIMESTAMPTZ DEFAULT NOW(), PRIMARY KEY (follower_id, followee_id), INDEX idx_followee (followee_id) ); ``` **Verify**: Query `SELECT COUNT(*) FROM information_schema.tables WHERE table_name IN ('users','posts','follows');` --> expected: `3` ### 2. Build the post ingestion pipeline When a user creates a post, write it to the Posts table, then publish an event to Kafka for async fan-out. [src2] ```python # post_service.py import json from kafka import KafkaProducer producer = KafkaProducer( bootstrap_servers=["kafka:9092"], value_serializer=lambda v: json.dumps(v).encode("utf-8"), ) def create_post(author_id: int, content: str, media_url: str = None) -> dict: # 1. Insert into database post = db.execute( "INSERT INTO posts (post_id, author_id, content, media_url) " "VALUES (nextval('post_id_seq'), %s, %s, %s) RETURNING *", (author_id, content, media_url), ) # 2. Publish to Kafka for fan-out producer.send( "post-created", key=str(author_id).encode(), value={"post_id": post["post_id"], "author_id": author_id}, ) return post ``` **Verify**: Publish a test post and check Kafka topic: `kafka-console-consumer --topic post-created --from-beginning` --> expected: JSON with post_id and author_id ### 3. Implement the fan-out service Kafka consumers read post-created events, look up the author's followers from the Social Graph Service, and push the post_id into each follower's Redis timeline. [src3] ```python # fanout_service.py import json import redis from kafka import KafkaConsumer r = redis.Redis(host="redis-cluster", port=6379, decode_responses=True) MAX_TIMELINE_SIZE = 800 consumer = KafkaConsumer( "post-created", bootstrap_servers=["kafka:9092"], group_id="fanout-workers", value_deserializer=lambda m: json.loads(m.decode("utf-8")), ) CELEBRITY_THRESHOLD = 500_000 def get_follower_ids(user_id: int) -> list[int]: """Query social graph service for follower list.""" return graph_service.get_followers(user_id) for message in consumer: event = message.value author_id = event["author_id"] post_id = event["post_id"] follower_ids = get_follower_ids(author_id) # Skip fan-out for celebrities -- handled at read time if len(follower_ids) > CELEBRITY_THRESHOLD: continue # Fan-out-on-write: push to each follower's timeline pipeline = r.pipeline() for fid in follower_ids: timeline_key = f"user:{fid}:timeline" pipeline.lpush(timeline_key, post_id) pipeline.ltrim(timeline_key, 0, MAX_TIMELINE_SIZE - 1) pipeline.execute() ``` **Verify**: After posting, check a follower's timeline: `redis-cli LRANGE user:42:timeline 0 9` --> expected: list containing the new post_id ### 4. Build the timeline read path The Timeline Service reads the precomputed Redis list, fetches full post objects (from cache or DB), and merges in celebrity tweets at serve time. [src2] [src3] ```python # timeline_service.py import redis import json r = redis.Redis(host="redis-cluster", port=6379, decode_responses=True) CELEBRITY_IDS = celebrity_service.get_celebrity_ids() # cached set def get_home_timeline(user_id: int, cursor: int = 0, limit: int = 20) -> list[dict]: # 1. Read precomputed timeline from Redis timeline_key = f"user:{user_id}:timeline" post_ids = r.lrange(timeline_key, cursor, cursor + limit - 1) # 2. Merge celebrity posts (fan-out-on-read) followed_celebrities = get_followed_celebrities(user_id, CELEBRITY_IDS) for celeb_id in followed_celebrities: recent = db.execute( "SELECT post_id FROM posts WHERE author_id = %s " "ORDER BY created_at DESC LIMIT %s", (celeb_id, limit), ) post_ids.extend([str(p["post_id"]) for p in recent]) # 3. Hydrate post objects (batch fetch from cache/DB) posts = hydrate_posts(post_ids) # 4. Sort by timestamp (or apply ranking) posts.sort(key=lambda p: p["created_at"], reverse=True) return posts[:limit] ``` **Verify**: `curl http://localhost:8080/api/v1/timeline?user_id=42&limit=10` --> expected: JSON array of 10 post objects sorted by recency ### 5. Add feed ranking (optional, for ranked feeds) Insert a ranking layer between hydration and response. Use a lightweight model scoring posts by engagement probability. [src4] [src5] ```python # ranking_service.py def rank_feed(user_id: int, candidate_posts: list[dict]) -> list[dict]: """Score and re-rank candidate posts for personalization.""" features = [] for post in candidate_posts: features.append({ "post_age_hours": hours_since(post["created_at"]), "author_follower_count": get_follower_count(post["author_id"]), "user_author_affinity": get_affinity_score(user_id, post["author_id"]), "post_like_count": post["like_count"], "post_has_media": 1 if post.get("media_url") else 0, }) # Score with pre-trained model (loaded at startup) scores = ranking_model.predict(features) # Attach scores and sort for post, score in zip(candidate_posts, scores): post["rank_score"] = float(score) candidate_posts.sort(key=lambda p: p["rank_score"], reverse=True) return candidate_posts ``` **Verify**: Compare ranked vs chronological output -- ranked feed should show higher-engagement posts first ### 6. Configure caching and CDN for media Set up pre-signed uploads to S3, configure CloudFront/Cloudflare CDN for delivery, and cache feed API responses at the edge. [src1] [src6] ```python # media_service.py import boto3 from botocore.config import Config s3 = boto3.client("s3", config=Config(signature_version="s3v4")) def generate_upload_url(user_id: int, filename: str) -> str: """Generate pre-signed URL for direct client upload.""" key = f"media/{user_id}/{filename}" url = s3.generate_presigned_url( "put_object", Params={"Bucket": "feed-media", "Key": key, "ContentType": "image/jpeg"}, ExpiresIn=3600, ) return url def get_media_cdn_url(key: str) -> str: """Return CDN URL for a stored media file.""" return f"https://cdn.example.com/{key}" ``` **Verify**: Upload a test image via pre-signed URL, then fetch via CDN URL -- expected: 200 OK with image content ## Code Examples ### Python: Feed Generation with Hybrid Fan-Out ```python # Input: user_id (int), cursor (int), limit (int) # Output: list of post dicts sorted by rank score import redis import json r = redis.Redis(host="redis-cluster", port=6379, decode_responses=True) CELEBRITY_THRESHOLD = 500_000 MAX_TIMELINE = 800 def generate_feed(user_id: int, cursor: int = 0, limit: int = 20) -> list[dict]: """Hybrid feed: precomputed timeline + celebrity merge + ranking.""" # Step 1: Read precomputed timeline (fan-out-on-write results) tl_key = f"user:{user_id}:timeline" post_ids = r.lrange(tl_key, cursor, cursor + limit * 2) # over-fetch for ranking # Step 2: Merge celebrity posts (fan-out-on-read) celeb_followees = get_followed_celebrities(user_id) for cid in celeb_followees: celeb_posts = r.lrange(f"user:{cid}:posts", 0, limit - 1) post_ids.extend(celeb_posts) # Step 3: Deduplicate post_ids = list(dict.fromkeys(post_ids)) # Step 4: Hydrate from cache, fallback to DB posts = batch_hydrate(post_ids) # Step 5: Rank and trim ranked = rank_feed(user_id, posts) return ranked[cursor:cursor + limit] ``` ### Python: Redis Timeline Caching > Full script: [python-redis-timeline-caching.py](scripts/python-redis-timeline-caching.py) (31 lines) ```python # Input: author_id (int), post_id (int), follower_ids (list[int]) # Output: None (side effect: updates Redis timelines) import redis r = redis.Redis(host="redis-cluster", port=6379, decode_responses=True) MAX_TIMELINE_SIZE = 800 # ... (see full script) ``` ## Anti-Patterns ### Wrong: Fan-out-on-write for all users regardless of follower count ```python # BAD -- celebrity with 10M followers causes 10M Redis writes per tweet def fan_out(author_id, post_id): followers = get_all_followers(author_id) # could be 10M+ for fid in followers: redis.lpush(f"user:{fid}:timeline", post_id) # Result: single tweet takes minutes to propagate, OOM risk ``` ### Correct: Hybrid fan-out with celebrity threshold ```python # GOOD -- skip fan-out for high-follower accounts, merge at read time CELEBRITY_THRESHOLD = 500_000 def fan_out(author_id, post_id): followers = get_all_followers(author_id) if len(followers) > CELEBRITY_THRESHOLD: return # handled by fan-out-on-read at serve time pipe = redis.pipeline(transaction=False) for fid in followers: pipe.lpush(f"user:{fid}:timeline", post_id) pipe.ltrim(f"user:{fid}:timeline", 0, 799) pipe.execute() ``` ### Wrong: Storing media files in the database ```python # BAD -- blobs in PostgreSQL cause table bloat and slow queries def save_post(content, image_bytes): db.execute( "INSERT INTO posts (content, image_data) VALUES (%s, %s)", (content, image_bytes), # 5MB image in a row! ) ``` ### Correct: Store media in object storage, reference by URL ```python # GOOD -- database stores only the URL reference def save_post(content, image_file): key = f"media/{uuid4()}.jpg" s3.upload_fileobj(image_file, "feed-media", key) cdn_url = f"https://cdn.example.com/{key}" db.execute( "INSERT INTO posts (content, media_url) VALUES (%s, %s)", (content, cdn_url), ) ``` ### Wrong: Unbounded timeline cache with no eviction ```python # BAD -- Redis memory grows without bound def add_to_timeline(user_id, post_id): redis.lpush(f"user:{user_id}:timeline", post_id) # No LTRIM, no TTL -- timeline grows forever # With 100M users, this will exhaust all Redis memory ``` ### Correct: Capped timeline with TTL and eviction ```python # GOOD -- bounded memory usage per user MAX_SIZE = 800 TTL = 7 * 24 * 3600 # 7 days def add_to_timeline(user_id, post_id): key = f"user:{user_id}:timeline" pipe = redis.pipeline() pipe.lpush(key, post_id) pipe.ltrim(key, 0, MAX_SIZE - 1) pipe.expire(key, TTL) pipe.execute() ``` ### Wrong: Synchronous fan-out blocking the post API ```python # BAD -- user waits for fan-out to complete before getting response @app.post("/api/posts") def create_post(request): post = db.insert_post(request.data) followers = get_followers(post.author_id) for fid in followers: # blocks for seconds with 100K followers redis.lpush(f"user:{fid}:timeline", post.id) return {"status": "ok", "post_id": post.id} # user waited 5+ seconds ``` ### Correct: Async fan-out via message queue ```python # GOOD -- return immediately, fan-out happens asynchronously @app.post("/api/posts") def create_post(request): post = db.insert_post(request.data) kafka.send("post-created", {"post_id": post.id, "author_id": post.author_id}) return {"status": "ok", "post_id": post.id} # instant response ``` ## Common Pitfalls - **Hot partition on celebrity user_id**: Kafka partitioning by author_id concentrates all celebrity activity on one partition. Fix: use a dedicated topic or random partitioning for high-follower accounts. [src3] - **Thundering herd on timeline rebuild**: When a popular user returns after inactivity, rebuilding their timeline from DB can spike load. Fix: rate-limit timeline rebuilds and use a circuit breaker. [src2] - **Stale celebrity tweets in merged feed**: Fan-out-on-read celebrity tweets may be fetched from a lagging read replica. Fix: query the primary for celebrity posts or cache recent celebrity posts in Redis with short TTL. [src3] - **Feed pagination drift**: New posts inserted during pagination cause duplicate or missing items. Fix: use cursor-based pagination (last_seen_post_id) instead of offset-based. [src1] - **Engagement count inconsistency**: Like/retweet counts updated in eventually-consistent caches show different numbers on refresh. Fix: use Redis INCR for counters with periodic DB reconciliation. [src6] - **Cache stampede on cold start**: First request after cache miss triggers expensive DB query while concurrent requests pile up. Fix: implement cache warming on deploy and use single-flight / lock-based cache population. [src7] - **Over-fetching in hydration**: Fetching full post objects (with all fields) when the feed only needs a subset. Fix: use a projection query or GraphQL-style field selection in the hydration layer. [src1] ## When to Use / When Not to Use | Use When | Don't Use When | Use Instead | |---|---|---| | Building a follow-graph-based feed (Twitter, Instagram, LinkedIn) | Content is purely algorithmic with no follow graph (TikTok For You page) | Recommendation engine with collaborative filtering | | DAU > 10K and feed latency matters (<200ms) | Small team project or MVP with <1K users | Simple SQL query with pagination | | Posts are short-lived and time-sensitive (news, social updates) | Content is long-form and rarely updated (blog, wiki) | CMS with traditional caching | | Real-time or near-real-time delivery is required | Batch-processed feeds updated hourly are acceptable | Batch ETL pipeline with scheduled materialization | | Multiple content types (text, images, video, polls) | Single content type with simple ordering | Standard list API with sorting | ## Important Caveats - Fan-out-on-write memory cost scales linearly with average follower count; at 500 avg followers and 100M users, expect ~40TB Redis if naively storing tweet IDs (mitigate with TTL + size caps + active-user-only caching) - Ranking models introduce latency (10-50ms per request); keep model inference on the critical path only if p99 latency budget allows, otherwise pre-rank in batch - GDPR/privacy regulations require the ability to delete a user's posts from all follower timelines within a defined time window; fan-out-on-write makes this harder (you must fan-out deletions too) - The hybrid fan-out threshold (e.g. 500K followers) is not universal; tune it based on your write throughput capacity and acceptable fan-out latency SLA - Multi-region deployment requires choosing between global consistency (higher latency) and regional feeds (users may see different content when traveling) ## Related Units - [Chat System Design](/software/system-design/chat-system/2026) - [Notification System Design](/software/system-design/notification-system/2026) - [Search Engine Design](/software/system-design/search-engine/2026)