Social Media Feed System Design

TL;DR

• Bottom line: A social media feed requires a hybrid fan-out architecture -- fan-out-on-write for normal users (precompute timelines into Redis) and fan-out-on-read for celebrity accounts (>500K followers) to avoid write amplification.
• Key tool/command: LPUSH user:{id}:timeline {tweet_id} (Redis list per user for O(1) feed reads)
• Watch out for: Unbounded fan-out for high-follower accounts -- a single celebrity tweet can generate millions of writes and lag the entire pipeline.
• Works with: Any stack; core patterns are language-agnostic. Redis + Kafka + PostgreSQL/Cassandra is the most battle-tested combination.

Constraints

• Fan-out-on-write must be disabled for users with >500K followers to avoid write amplification storms
• Timeline cache must have a TTL and size cap (e.g. 800 tweets per user) to bound memory usage
• Never store media blobs in the primary database; use object storage (S3/GCS) with CDN delivery
• Feed ranking models require A/B testing infrastructure before production deployment
• Eventual consistency is acceptable for feed delivery (seconds-level lag), but like/reply counts need at-least-once delivery guarantees

Quick Reference

Decision Tree

START
|-- Expected DAU?
|   |-- <10K users
|   |   --> Simple pull-based feed: query DB at read time, paginate with cursor
|   |-- 10K - 1M users
|   |   --> Fan-out-on-write for all users
|   |   --> Precompute timelines into Redis on every post
|   |-- 1M - 100M users
|   |   |-- Any user with >500K followers?
|   |   |   |-- YES --> Hybrid: fan-out-on-write for normal, fan-out-on-read for celebrities
|   |   |   |-- NO --> Pure fan-out-on-write with Redis Cluster
|   |   |-- Feed type?
|   |   |   |-- Chronological --> Sort by timestamp in Redis sorted set
|   |   |   |-- Ranked --> Add Ranking Service with ML model
|   |   |   |-- Hybrid --> Chronological base + lightweight re-ranking
|   |-- >100M users
|       --> Full hybrid fan-out + ML ranking pipeline
|       --> Multi-region deployment with geo-sharding
|       --> Edge caching for feed responses

Step-by-Step Guide

1. Define the data model and API contracts

Design the core entities: Users, Posts, Follows, Likes, Comments. Define REST or gRPC endpoints for create-post, get-timeline, follow/unfollow. [src1]

CREATE TABLE users (
    user_id    BIGINT PRIMARY KEY,
    username   VARCHAR(64) UNIQUE NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE posts (
    post_id    BIGINT PRIMARY KEY,
    author_id  BIGINT NOT NULL REFERENCES users(user_id),
    content    TEXT NOT NULL,
    media_url  TEXT,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    like_count INT DEFAULT 0
);

CREATE TABLE follows (
    follower_id BIGINT NOT NULL,
    followee_id BIGINT NOT NULL,
    created_at  TIMESTAMPTZ DEFAULT NOW(),
    PRIMARY KEY (follower_id, followee_id)
);

Verify: SELECT COUNT(*) FROM information_schema.tables WHERE table_name IN ('users','posts','follows'); --> expected: 3

2. Build the post ingestion pipeline

When a user creates a post, write it to the Posts table, then publish an event to Kafka for async fan-out. [src2]

producer = KafkaProducer(bootstrap_servers=["kafka:9092"])

def create_post(author_id, content, media_url=None):
    post = db.insert_post(author_id, content, media_url)
    producer.send("post-created",
        key=str(author_id).encode(),
        value=json.dumps({"post_id": post["post_id"], "author_id": author_id}).encode()
    )
    return post

Verify: kafka-console-consumer --topic post-created --from-beginning --> expected: JSON with post_id

3. Implement the fan-out service

Kafka consumers read post-created events, look up followers, and push post_id into each follower's Redis timeline. Skip fan-out for celebrities. [src3]

CELEBRITY_THRESHOLD = 500_000

for message in consumer:
    event = message.value
    followers = get_followers(event["author_id"])
    if len(followers) > CELEBRITY_THRESHOLD:
        continue  # handled at read time
    pipe = redis.pipeline(transaction=False)
    for fid in followers:
        pipe.lpush(f"user:{fid}:timeline", event["post_id"])
        pipe.ltrim(f"user:{fid}:timeline", 0, 799)
    pipe.execute()

Verify: redis-cli LRANGE user:42:timeline 0 9 --> expected: list with new post_id

4. Build the timeline read path

Read precomputed Redis list, hydrate post objects, merge celebrity tweets at serve time. [src2] [src3]

def get_home_timeline(user_id, cursor=0, limit=20):
    post_ids = redis.lrange(f"user:{user_id}:timeline", cursor, cursor + limit - 1)
    for celeb_id in get_followed_celebrities(user_id):
        celeb_posts = db.query_recent_posts(celeb_id, limit)
        post_ids.extend(celeb_posts)
    posts = hydrate_posts(list(dict.fromkeys(post_ids)))
    posts.sort(key=lambda p: p["created_at"], reverse=True)
    return posts[:limit]

Verify: curl /api/v1/timeline?user_id=42&limit=10 --> expected: JSON array of posts

5. Add feed ranking

Insert a ranking layer scoring posts by engagement probability using a lightweight ML model. [src4] [src5]

def rank_feed(user_id, candidates):
    features = extract_features(user_id, candidates)
    scores = ranking_model.predict(features)
    for post, score in zip(candidates, scores):
        post["rank_score"] = float(score)
    candidates.sort(key=lambda p: p["rank_score"], reverse=True)
    return candidates

Verify: Compare ranked vs chronological output -- ranked feed should prioritize higher-engagement posts

6. Configure media storage and CDN

Set up pre-signed uploads to S3, configure CDN for delivery, cache feed API responses at the edge. [src1] [src6]

def generate_upload_url(user_id, filename):
    key = f"media/{user_id}/{filename}"
    return s3.generate_presigned_url("put_object",
        Params={"Bucket": "feed-media", "Key": key},
        ExpiresIn=3600
    )

Verify: Upload test image via pre-signed URL, fetch via CDN --> expected: 200 OK

Code Examples

Python: Feed Generation with Hybrid Fan-Out

# Input:  user_id (int), cursor (int), limit (int)
# Output: list of post dicts sorted by rank score

import redis

r = redis.Redis(host="redis-cluster", port=6379, decode_responses=True)

def generate_feed(user_id, cursor=0, limit=20):
    """Hybrid feed: precomputed timeline + celebrity merge + ranking."""
    tl_key = f"user:{user_id}:timeline"
    post_ids = r.lrange(tl_key, cursor, cursor + limit * 2)
    celeb_followees = get_followed_celebrities(user_id)
    for cid in celeb_followees:
        celeb_posts = r.lrange(f"user:{cid}:posts", 0, limit - 1)
        post_ids.extend(celeb_posts)
    post_ids = list(dict.fromkeys(post_ids))
    posts = batch_hydrate(post_ids)
    ranked = rank_feed(user_id, posts)
    return ranked[cursor:cursor + limit]

Python: Redis Timeline Caching

# Input:  author_id (int), post_id (int), follower_ids (list[int])
# Output: None (updates Redis timelines)

import redis

r = redis.Redis(host="redis-cluster", port=6379, decode_responses=True)
MAX_TIMELINE_SIZE = 800
TIMELINE_TTL = 7 * 24 * 3600  # 7 days

def fan_out_to_timelines(author_id, post_id, follower_ids):
    pipe = r.pipeline(transaction=False)
    for fid in follower_ids:
        key = f"user:{fid}:timeline"
        pipe.lpush(key, post_id)
        pipe.ltrim(key, 0, MAX_TIMELINE_SIZE - 1)
        pipe.expire(key, TIMELINE_TTL)
    pipe.execute()

Anti-Patterns

Wrong: Fan-out-on-write for all users regardless of follower count

# BAD -- celebrity with 10M followers causes 10M Redis writes per tweet
def fan_out(author_id, post_id):
    followers = get_all_followers(author_id)  # could be 10M+
    for fid in followers:
        redis.lpush(f"user:{fid}:timeline", post_id)
    # Result: single tweet takes minutes, OOM risk

Correct: Hybrid fan-out with celebrity threshold

# GOOD -- skip fan-out for high-follower accounts, merge at read time
CELEBRITY_THRESHOLD = 500_000

def fan_out(author_id, post_id):
    followers = get_all_followers(author_id)
    if len(followers) > CELEBRITY_THRESHOLD:
        return  # handled at read time
    pipe = redis.pipeline(transaction=False)
    for fid in followers:
        pipe.lpush(f"user:{fid}:timeline", post_id)
        pipe.ltrim(f"user:{fid}:timeline", 0, 799)
    pipe.execute()

Wrong: Storing media files in the database

# BAD -- blobs in PostgreSQL cause table bloat and slow queries
def save_post(content, image_bytes):
    db.execute("INSERT INTO posts (content, image_data) VALUES (%s, %s)",
        (content, image_bytes))  # 5MB image in a row!

Correct: Store media in object storage, reference by URL

# GOOD -- database stores only the URL reference
def save_post(content, image_file):
    key = f"media/{uuid4()}.jpg"
    s3.upload_fileobj(image_file, "feed-media", key)
    cdn_url = f"https://cdn.example.com/{key}"
    db.execute("INSERT INTO posts (content, media_url) VALUES (%s, %s)",
        (content, cdn_url))

Wrong: Unbounded timeline cache with no eviction

# BAD -- Redis memory grows without bound
def add_to_timeline(user_id, post_id):
    redis.lpush(f"user:{user_id}:timeline", post_id)
    # No LTRIM, no TTL -- grows forever, exhausts Redis memory

Correct: Capped timeline with TTL and eviction

# GOOD -- bounded memory usage per user
def add_to_timeline(user_id, post_id):
    key = f"user:{user_id}:timeline"
    pipe = redis.pipeline()
    pipe.lpush(key, post_id)
    pipe.ltrim(key, 0, 799)
    pipe.expire(key, 7 * 24 * 3600)
    pipe.execute()

Wrong: Synchronous fan-out blocking the post API

# BAD -- user waits for fan-out to complete
@app.post("/api/posts")
def create_post(request):
    post = db.insert_post(request.data)
    for fid in get_followers(post.author_id):  # blocks for seconds
        redis.lpush(f"user:{fid}:timeline", post.id)
    return {"status": "ok"}  # user waited 5+ seconds

Correct: Async fan-out via message queue

# GOOD -- return immediately, fan-out asynchronously
@app.post("/api/posts")
def create_post(request):
    post = db.insert_post(request.data)
    kafka.send("post-created", {"post_id": post.id, "author_id": post.author_id})
    return {"status": "ok"}  # instant response

Common Pitfalls

• Hot partition on celebrity user_id: Kafka partitioning by author_id concentrates all celebrity activity on one partition. Fix: use dedicated topic or random partitioning for high-follower accounts. [src3]
• Thundering herd on timeline rebuild: When a popular user returns after inactivity, rebuilding their timeline from DB spikes load. Fix: rate-limit rebuilds and use a circuit breaker. [src2]
• Stale celebrity tweets in merged feed: Fan-out-on-read celebrity tweets fetched from lagging read replica. Fix: query primary for celebrity posts or cache with short TTL. [src3]
• Feed pagination drift: New posts during pagination cause duplicates or missing items. Fix: use cursor-based pagination (last_seen_post_id) instead of offset. [src1]
• Engagement count inconsistency: Like/retweet counts in caches show different numbers on refresh. Fix: use Redis INCR for counters with periodic DB reconciliation. [src6]
• Cache stampede on cold start: First request after cache miss triggers expensive DB query while concurrent requests pile up. Fix: cache warming on deploy + single-flight cache population. [src7]
• Over-fetching in hydration: Fetching full post objects when the feed only needs a subset of fields. Fix: use projection queries or field selection in hydration. [src1]

When to Use / When Not to Use

Important Caveats

• Fan-out-on-write memory cost scales linearly with average follower count; at 500 avg followers and 100M users, expect ~40TB Redis if naively storing tweet IDs (mitigate with TTL + size caps + active-user-only caching)
• Ranking models introduce latency (10-50ms per request); keep model inference on the critical path only if p99 latency budget allows
• GDPR/privacy regulations require the ability to delete a user's posts from all follower timelines within a defined time window; fan-out-on-write makes this harder (you must fan-out deletions too)
• The hybrid fan-out threshold (e.g. 500K followers) is not universal; tune it based on your write throughput capacity and acceptable fan-out latency SLA
• Multi-region deployment requires choosing between global consistency (higher latency) and regional feeds (users may see different content when traveling)