--- # === IDENTITY === id: software/system-design/url-shortener/2026 canonical_question: "How do I design a URL shortener at scale?" aliases: - "How to build a URL shortening service like Bitly or TinyURL?" - "URL shortener system design interview" - "Scalable short link service architecture" - "Design a URL shortener with billions of links" entity_type: software_reference domain: software > system-design > url_shortener region: global jurisdiction: global temporal_scope: 2020-2026 # === VERIFICATION === last_verified: 2026-02-23 confidence: 0.93 version: 1.0 first_published: 2026-02-23 # === TEMPORAL VALIDITY === temporal_validity: status: stable last_breaking_change: null next_review: 2026-08-22 change_sensitivity: low # === CONSTRAINTS === constraints: - "Short codes must be at least 6-7 characters in base62 to support billions of unique URLs (62^7 = 3.5 trillion)" - "Read-to-write ratio is ~100:1; optimize the redirect (read) path first" - "Never use HTTP 301 (permanent redirect) if you need click analytics — browsers cache 301s and skip your server" - "URL validation and malware scanning must happen at write time, not redirect time" - "Cache invalidation must propagate to all edge nodes when a short URL is deleted or updated" # === SKIP CONDITIONS === skip_this_unit_if: - condition: "Need to design a paste/content-sharing service (Pastebin)" use_instead: "software/system-design/pastebin/2026" - condition: "Need a CDN or content delivery architecture" use_instead: "software/system-design/cdn-design/2026" # === AGENT HINTS === inputs_needed: - key: "expected_scale" question: "What is the expected scale? (URLs created per day and redirects per day)" type: choice options: ["<100K URLs/day (small)", "100K-10M URLs/day (medium)", ">10M URLs/day (large)"] - key: "analytics_required" question: "Do you need click analytics (geo, referrer, device)?" type: choice options: ["yes", "no"] - key: "custom_aliases" question: "Do you need custom/vanity short URLs?" type: choice options: ["yes", "no"] # === DISTRIBUTION === canonical_source: "https://knowledgelib.io/software/system-design/url-shortener/2026" suggested_citation: "Source: knowledgelib.io — AI Knowledge Library (verified 2026-02-23)" # === RELATED UNITS === related_kos: depends_on: [] related_to: - id: "software/system-design/rate-limiter/2026" label: "Rate Limiter System Design" solves: [] alternative_to: [] often_confused_with: - id: "software/system-design/pastebin/2026" label: "Pastebin System Design" # === SOURCES (7 authoritative sources) === sources: - id: src1 title: "URL Shortening System Design" author: systemdesign.one url: https://systemdesign.one/url-shortening-system-design/ type: technical_blog published: 2024-06-15 reliability: high - id: src2 title: "Design a URL Shortener — System Design Interview" author: AlgoMaster url: https://blog.algomaster.io/p/design-a-url-shortener type: technical_blog published: 2024-09-10 reliability: high - id: src3 title: "How to Design a URL Shortener Service" author: Design Gurus url: https://www.designgurus.io/blog/url-shortening type: technical_blog published: 2024-07-20 reliability: high - id: src4 title: "Design a URL Shortener Like Bit.ly" author: Hello Interview url: https://www.hellointerview.com/learn/system-design/problem-breakdowns/bitly type: technical_blog published: 2024-08-05 reliability: high - id: src5 title: "Building a Scalable URL Shortener with Distributed Caching Using Redis" author: freeCodeCamp url: https://www.freecodecamp.org/news/build-a-scalable-url-shortener-with-distributed-caching-using-redis/ type: technical_blog published: 2024-11-12 reliability: moderate_high - id: src6 title: "URL Shortener System Design — GeeksforGeeks" author: GeeksforGeeks url: https://www.geeksforgeeks.org/system-design-url-shortening-service/ type: community_resource published: 2024-10-01 reliability: moderate_high - id: src7 title: "System Design Deep Dive: Designing a URL Shortener with Base62" author: Design Gurus url: https://designgurus.substack.com/p/system-design-deep-dive-designing type: technical_blog published: 2024-05-18 reliability: moderate_high --- # URL Shortener System Design at Scale ## TL;DR - **Bottom line**: A URL shortener maps long URLs to short 6-7 character codes using base62 encoding over unique IDs, stores mappings in a NoSQL database, and uses multi-layer caching (CDN + Redis) to serve redirects at <50ms latency with a 100:1 read-to-write ratio. - **Key tool/command**: `base62_encode(unique_id)` to generate short codes; Redis for caching hot redirects - **Watch out for**: Using HTTP 301 (permanent redirect) when you need analytics -- browsers cache 301s permanently, so your server never sees repeat clicks. - **Works with**: Any language/stack. Core components: load balancer, app servers, Redis/Memcached, Cassandra/DynamoDB/PostgreSQL, Kafka for analytics. ## Constraints - Short codes must be at least 6-7 characters in base62 to support billions of unique URLs (62^7 = 3.5 trillion combinations) - Read-to-write ratio is approximately 100:1; the redirect (read) path must be optimized first - Never use HTTP 301 (permanent redirect) if you need click analytics — browsers cache 301s and skip your server on subsequent visits - URL validation and malware scanning (e.g., Google Safe Browsing API) must happen at write time, not redirect time, to avoid adding latency to redirects - Cache invalidation must propagate to all edge/cache nodes when a short URL is deleted, updated, or expires ## Quick Reference | Component | Role | Technology Options | Scaling Strategy | |---|---|---|---| | Load Balancer | Distributes traffic across app servers | AWS ALB, Nginx, HAProxy | Geo-DNS routing + multiple LBs per region | | API Gateway | Rate limiting, auth, request routing | Kong, AWS API Gateway | Horizontal scaling behind LB | | URL Creation Service | Accepts long URLs, generates short codes | Go, Java, Node.js | Stateless; scale horizontally | | Redirect Service | Resolves short code to long URL, returns 302 | Go, Rust (low latency) | Separate from write path; scale independently | | ID Generator | Produces globally unique IDs for base62 encoding | Snowflake, Zookeeper counter ranges | Pre-allocate ID ranges per node to avoid coordination | | Cache Layer | Stores hot short-to-long URL mappings | Redis Cluster, Memcached | LRU eviction; 20% of URLs serve 80% of traffic | | Primary Database | Persistent URL mapping storage | Cassandra, DynamoDB, PostgreSQL | Hash-based sharding on short_code | | CDN / Edge Cache | Caches redirects at edge locations | Cloudflare, CloudFront | Cache 302 responses with short TTL (5-60 min) | | Analytics Pipeline | Captures click events asynchronously | Kafka + Flink/Spark, ClickHouse | Decouple from redirect path; process in batches | | Cleanup Service | Removes expired URLs | Cron job / background worker | Runs during low-traffic windows; lazy deletion on access | | Abuse Prevention | Rate limiting, malware scanning | Redis (token bucket), Google Safe Browsing | Per-IP and per-API-key rate limits | | Monitoring | System health, latency tracking | Prometheus + Grafana, Datadog | Alert on p99 redirect latency > 100ms | ## Decision Tree > Full script: [decision-tree.txt](scripts/decision-tree.txt) (29 lines) ``` START |-- Expected write volume? | |-- <1K URLs/day (hobby/internal)? | | |-- Use single PostgreSQL instance + in-memory cache | | |-- Auto-increment ID + base62 encode # ... (see full script) ``` ## Step-by-Step Guide ### 1. Define capacity requirements Estimate write and read QPS, storage, and bandwidth based on expected traffic. [src1] ``` Given: 100M URLs created per month Write QPS: 100M / (30 * 24 * 3600) = ~40 writes/sec Read QPS: 40 * 100 = ~4,000 reads/sec (100:1 ratio) Peak: 10x average = 400 writes/sec, 40,000 reads/sec Storage per URL: ~500 bytes (short_code + long_url + metadata) Annual storage: 100M * 12 * 500 bytes = ~600 GB/year 5-year storage: ~3 TB (before replication) Cache (80-20 rule): 20% of daily reads * avg URL size = 0.2 * (4000 * 86400) * 500 bytes = ~34 GB ``` **Verify**: Confirm your peak QPS estimate accounts for viral spikes (10-50x normal traffic). ### 2. Design the database schema Choose a key-value-friendly schema optimized for short_code lookups. [src2] ```sql -- Primary URL mappings (or use NoSQL equivalent) CREATE TABLE url_mappings ( short_code VARCHAR(7) PRIMARY KEY, -- base62 encoded long_url TEXT NOT NULL, created_at TIMESTAMP DEFAULT NOW(), expires_at TIMESTAMP, user_id BIGINT, click_count BIGINT DEFAULT 0 ); -- Index for deduplication (optional: same long URL -> same short code) CREATE INDEX idx_long_url ON url_mappings (long_url); -- Analytics events (or use Kafka -> ClickHouse) CREATE TABLE click_events ( event_id BIGINT PRIMARY KEY, short_code VARCHAR(7), clicked_at TIMESTAMP, referrer TEXT, user_agent TEXT, country VARCHAR(2), ip_hash VARCHAR(64) ); ``` **Verify**: Ensure `short_code` is the primary key / partition key for O(1) lookups. ### 3. Implement short code generation Use a unique ID generator + base62 encoding for collision-free codes. [src7] ```python import string ALPHABET = string.digits + string.ascii_lowercase + string.ascii_uppercase # 62 chars def base62_encode(num: int) -> str: """Convert a positive integer to a base62 string.""" if num == 0: return ALPHABET[0] result = [] while num > 0: num, remainder = divmod(num, 62) result.append(ALPHABET[remainder]) return ''.join(reversed(result)) def base62_decode(code: str) -> int: """Convert a base62 string back to an integer.""" num = 0 for char in code: num = num * 62 + ALPHABET.index(char) return num # Example: unique_id=123456789 -> short_code="8M0kX" print(base62_encode(123456789)) # "8M0kX" ``` **Verify**: `base62_decode(base62_encode(n)) == n` for any positive integer `n`. ### 4. Set up caching with Redis Cache frequently accessed URL mappings to serve redirects from memory. [src5] ```python import redis r = redis.Redis(host='redis-cluster', port=6379, decode_responses=True) CACHE_TTL = 3600 # 1 hour def get_long_url(short_code: str) -> str | None: # 1. Check cache first cached = r.get(f"url:{short_code}") if cached: return cached # 2. Cache miss -> query database long_url = db_lookup(short_code) # your DB query here if long_url: r.setex(f"url:{short_code}", CACHE_TTL, long_url) return long_url ``` **Verify**: Monitor cache hit ratio — target >90%. `redis-cli INFO stats | grep hit_rate` ### 5. Build the redirect service Handle incoming short URL requests and issue HTTP 302 redirects. [src1] ```python from fastapi import FastAPI, HTTPException from fastapi.responses import RedirectResponse app = FastAPI() @app.get("/{short_code}") async def redirect(short_code: str): long_url = get_long_url(short_code) if not long_url: raise HTTPException(status_code=404, detail="Short URL not found") # Publish click event asynchronously (don't block redirect) publish_click_event(short_code, request) # 302 = temporary redirect (server sees every click for analytics) return RedirectResponse(url=long_url, status_code=302) ``` **Verify**: `curl -I https://short.url/abc123` returns `HTTP/1.1 302` with `Location:` header. ### 6. Add analytics pipeline Decouple analytics from the redirect path using a message queue. [src1] ```python from kafka import KafkaProducer import json producer = KafkaProducer( bootstrap_servers='kafka:9092', value_serializer=lambda v: json.dumps(v).encode('utf-8') ) def publish_click_event(short_code, request): event = { "short_code": short_code, "timestamp": datetime.utcnow().isoformat(), "referrer": request.headers.get("referer", ""), "user_agent": request.headers.get("user-agent", ""), "ip_country": geoip_lookup(request.client.host), } producer.send("click-events", value=event) # Non-blocking: redirect returns immediately ``` **Verify**: Check Kafka consumer lag: `kafka-consumer-groups.sh --describe --group analytics-consumer` ## Code Examples ### Python: Base62 Encoding with Collision-Free ID Generation > Full script: [python-base62-encoding-with-collision-free-id-gene.py](scripts/python-base62-encoding-with-collision-free-id-gene.py) (41 lines) ```python # Input: A unique integer ID from your ID generator # Output: A 6-7 character URL-safe short code import string import time import threading # ... (see full script) ``` ### Node.js: Redirect Service with Redis Caching > Full script: [node-js-redirect-service-with-redis-caching.js](scripts/node-js-redirect-service-with-redis-caching.js) (38 lines) ```javascript // Input: HTTP GET request with short code in URL path // Output: HTTP 302 redirect to original long URL const express = require('express'); // express@4.18 const Redis = require('ioredis'); // ioredis@5.3 const { Kafka } = require('kafkajs'); // kafkajs@2.2 # ... (see full script) ``` ## Anti-Patterns ### Wrong: Using MD5/SHA hash of the URL and truncating ```python # BAD -- truncated hashes cause collisions at scale import hashlib, base64 def shorten(long_url): md5 = hashlib.md5(long_url.encode()).digest() return base64.urlsafe_b64encode(md5[:6]).decode()[:7] # At ~10M URLs, collision probability exceeds 1% # No way to recover from collision without re-hashing ``` ### Correct: Use a unique ID generator + base62 ```python # GOOD -- unique IDs guarantee zero collisions def shorten(long_url, id_generator, db): unique_id = id_generator.next_id() # globally unique short_code = base62_encode(unique_id) db.insert(short_code, long_url) return short_code # Collision-free by construction # ID generator handles distributed uniqueness ``` ### Wrong: Checking the database on every redirect ```python # BAD -- every redirect hits the database @app.get("/{code}") async def redirect(code: str): row = await db.query("SELECT long_url FROM urls WHERE code = $1", [code]) return RedirectResponse(url=row.long_url, status_code=302) # At 40K reads/sec, the database becomes the bottleneck # p99 latency climbs to 200-500ms ``` ### Correct: Cache-aside pattern with Redis ```python # GOOD -- cache handles >90% of reads @app.get("/{code}") async def redirect(code: str): long_url = await redis.get(f"url:{code}") # ~1ms if not long_url: row = await db.query("...", [code]) # ~10-50ms if row: await redis.setex(f"url:{code}", 3600, row.long_url) long_url = row.long_url if row else None return RedirectResponse(url=long_url, status_code=302) # 90%+ cache hit rate -> avg latency <5ms ``` ### Wrong: Using HTTP 301 when you need analytics ```python # BAD -- 301 means browser caches the redirect permanently return RedirectResponse(url=long_url, status_code=301) # After the first click, the browser goes directly to the long URL # Your server never sees subsequent clicks # Analytics data is incomplete and misleading ``` ### Correct: Use HTTP 302 for trackable redirects ```python # GOOD -- 302 means browser asks your server every time return RedirectResponse(url=long_url, status_code=302) # Every click passes through your server # Full analytics: click count, referrer, geo, device # Trade-off: slightly higher server load (offset by CDN caching with short TTL) ``` ### Wrong: Synchronous analytics in the redirect path ```python # BAD -- analytics write blocks the redirect response @app.get("/{code}") async def redirect(code: str): long_url = get_long_url(code) # This write adds 20-100ms to every redirect await db.execute("INSERT INTO clicks ...", [code, datetime.now()]) await db.execute("UPDATE urls SET click_count = click_count + 1 ...", [code]) return RedirectResponse(url=long_url, status_code=302) ``` ### Correct: Async analytics via message queue ```python # GOOD -- fire-and-forget to Kafka, redirect returns immediately @app.get("/{code}") async def redirect(code: str): long_url = get_long_url(code) # Non-blocking: Kafka producer buffers and sends in background kafka_producer.send("click-events", {"code": code, "ts": time.time()}) return RedirectResponse(url=long_url, status_code=302) # Redirect latency unaffected by analytics # Kafka consumers process events asynchronously into ClickHouse/HDFS ``` ## Common Pitfalls - **Hash collision at scale**: MD5/SHA truncation causes collisions once you exceed millions of URLs. Fix: use a unique ID generator (Snowflake, Zookeeper counter ranges) + base62 encoding instead of hashing. [src2] - **Single-point-of-failure ID generator**: A single auto-increment counter becomes a bottleneck and SPOF. Fix: use distributed ID generation with pre-allocated ranges (e.g., node 1 gets IDs 1-10000, node 2 gets 10001-20000) via Zookeeper. [src1] - **Cache stampede on popular URLs**: When a cache entry expires, thousands of simultaneous requests hit the database. Fix: use cache warming, staggered TTLs (TTL + random jitter), or request coalescing (single-flight pattern). [src5] - **No expiration/cleanup strategy**: Orphaned URLs consume storage indefinitely. Fix: set default expiration (e.g., 2 years), run a cleanup service during off-peak hours, and implement lazy deletion (check expiry on access). [src3] - **Ignoring abuse vectors**: Short URLs can mask phishing/malware destinations. Fix: validate URLs at creation time using Google Safe Browsing API, implement per-IP and per-API-key rate limiting (e.g., 100 creates/hour). [src6] - **Predictable short codes**: Sequential base62 codes let attackers enumerate all URLs. Fix: apply a reversible bit-shuffle or XOR cipher to the ID before base62 encoding, or use a pre-generated random key pool. [src1] - **Not separating read and write paths**: Using the same service for creation and redirection means a traffic spike in redirects degrades creation. Fix: deploy separate microservices for URL creation vs. redirect, with independent scaling. [src4] - **302 redirect without CDN caching**: Every redirect hits your origin servers. Fix: set `Cache-Control: public, max-age=300` on 302 responses so CDN edges cache redirects for 5 minutes, reducing origin load by 60-80%. [src1] ## Diagnostic Commands ```bash # Check Redis cache hit rate redis-cli INFO stats | grep -E "keyspace_hits|keyspace_misses" # Monitor redirect latency (p50, p95, p99) curl -w "time_total: %{time_total}s\n" -o /dev/null -s https://short.url/abc123 # Check Kafka consumer lag for analytics pipeline kafka-consumer-groups.sh --bootstrap-server kafka:9092 \ --describe --group analytics-consumer # Verify database connection pool usage SELECT count(*) FROM pg_stat_activity WHERE state = 'active'; # Check URL mapping count and storage size SELECT count(*) AS total_urls, pg_size_pretty(pg_total_relation_size('url_mappings')) AS storage FROM url_mappings; # Test short code generation uniqueness (should print 100000 unique codes) python -c " from collections import Counter codes = [base62_encode(i) for i in range(100000)] dupes = {k: v for k, v in Counter(codes).items() if v > 1} print(f'Unique: {len(set(codes))}, Duplicates: {len(dupes)}') " ``` ## When to Use / When Not to Use | Use When | Don't Use When | Use Instead | |---|---|---| | Sharing long URLs in space-constrained contexts (SMS, tweets, QR codes) | URLs are already short or internal-only | Direct links with no shortening | | You need click analytics (geo, referrer, device, time) | Privacy regulations prohibit redirect tracking (some EU contexts) | Direct links with server-side analytics | | Marketing campaigns requiring branded short domains | You need content hosting/sharing (not just redirects) | Pastebin or object storage service | | You need A/B testing via dynamic redirect targets | URL mappings change frequently (more write-heavy than read-heavy) | Feature flag service or API gateway routing | | QR code generation (shorter URLs = simpler QR patterns) | You only need vanity domains without analytics | DNS CNAME record or reverse proxy | ## Important Caveats - Base62 encoding of sequential IDs produces predictable short codes; attackers can enumerate URLs by incrementing. Apply a reversible bit-shuffle, XOR cipher, or use pre-generated random key pools to mitigate. - HTTP 301 vs 302 is the single most consequential architectural decision: 301 reduces server load but makes analytics impossible; 302 enables full analytics but increases origin traffic. CDN edge caching with short TTLs (5-60 min) is the standard compromise. - At Bitly-scale (billions of redirects/day), the analytics pipeline (Kafka -> Flink/Spark -> ClickHouse) often costs more to operate than the redirect infrastructure itself. Budget accordingly. - URL shorteners are frequently abused for phishing. Integrating Google Safe Browsing or similar malware detection at URL creation time is not optional for any public-facing service. - Database choice matters less than caching strategy. PostgreSQL, Cassandra, and DynamoDB all work at scale when 90%+ of reads are served from Redis/Memcached. Pick what your team knows. ## Related Units - [Rate Limiter System Design](/software/system-design/rate-limiter/2026) - [Consistent Hashing](/software/patterns/consistent-hashing/2026) - [Distributed ID Generation](/software/system-design/distributed-id-generation/2026)