--- # === IDENTITY === id: software/system-design/recommendation-engine/2026 canonical_question: "How do I design a recommendation engine?" aliases: - "recommendation system architecture" - "build a recommender system from scratch" - "collaborative filtering system design" - "content-based recommendation engine" - "hybrid recommendation system architecture" - "personalization engine design" - "ML recommendation pipeline architecture" - "design a recommendation service at scale" entity_type: software_reference domain: software > system-design > recommendation_engine region: global jurisdiction: global temporal_scope: 2020-2026 # === VERIFICATION === last_verified: 2026-02-23 confidence: 0.90 freshness: quarterly version: 1.0 first_published: 2026-02-23 # === TEMPORAL VALIDITY === temporal_validity: status: stable last_breaking_change: null next_review: 2026-08-22 change_sensitivity: medium # === CONSTRAINTS === constraints: - "Never train on raw user data without anonymization and consent — GDPR/CCPA require explicit opt-in for behavioral profiling and right-to-explanation for automated decisions" - "Always separate retrieval (candidate generation) from ranking — a single-stage model cannot serve low-latency recommendations over catalogs >100K items" - "Never serve recommendations without a fallback strategy — cold-start users and items must get non-personalized (popularity-based) results, not empty responses" - "Feature store must guarantee consistency between training and serving — training-serving skew is the #1 silent killer of recommendation quality" - "A/B test every model change against business metrics (revenue, engagement), not just offline metrics (NDCG, recall) — offline gains frequently do not translate to online improvements" # === SKIP CONDITIONS === skip_this_unit_if: - condition: "User needs search/ranking for explicit keyword queries, not personalized recommendations" use_instead: "software/system-design/search-engine/2026" - condition: "User wants to implement a simple rule-based 'related items' feature without ML" use_instead: "Use co-occurrence counting or manual curation — no ML pipeline needed" - condition: "User needs ad ranking or auction-based placement, not content recommendations" use_instead: "software/system-design/ad-ranking-system/2026" # === AGENT HINTS === inputs_needed: - key: catalog_size question: "How many items are in the catalog?" type: choice options: ["Small (<10K)", "Medium (10K-1M)", "Large (1M-100M)", "Massive (>100M)"] - key: interaction_type question: "What type of user interactions do you have?" type: choice options: ["Implicit only (clicks, views, time-spent)", "Explicit (ratings, likes)", "Both implicit and explicit", "No historical data yet (cold start)"] - key: latency_requirement question: "What is the acceptable p99 latency for serving recommendations?" type: choice options: ["<50ms (real-time feed)", "<200ms (page load)", "<1s (email/push)", "Batch (offline, no latency constraint)"] # === DISTRIBUTION === canonical_source: "https://knowledgelib.io/software/system-design/recommendation-engine/2026" suggested_citation: "Source: knowledgelib.io — AI Knowledge Library (verified 2026-02-23)" # === RELATED UNITS === related_kos: depends_on: - id: "software/system-design/feature-store/2026" label: "Feature Store Architecture" related_to: - id: "software/system-design/search-engine/2026" label: "Search Engine Design" - id: "software/system-design/ad-ranking-system/2026" label: "Ad Ranking System Design" - id: "software/system-design/real-time-analytics/2026" label: "Real-Time Analytics Pipeline" solves: - id: "software/system-design/personalization-service/2026" label: "Personalization Service Architecture" alternative_to: - id: "software/system-design/rule-based-recommendations/2026" label: "Rule-Based Recommendation System" often_confused_with: - id: "software/system-design/search-engine/2026" label: "Search Engine — user-initiated queries vs. system-initiated personalization" # === SOURCES (7 authoritative sources) === # Types: official_docs, technical_blog, rfc_spec, academic_paper, community_resource, industry_report # Reliability: high, moderate_high, moderate, moderate_low, low, authoritative sources: - id: src1 title: "System Architectures for Personalization and Recommendation" author: Netflix Technology Blog url: https://netflixtechblog.com/system-architectures-for-personalization-and-recommendation-e081aa94b5d8 type: technical_blog published: 2013-03-27 reliability: authoritative - id: src2 title: "System Design for Recommendations and Search" author: Eugene Yan url: https://eugeneyan.com/writing/system-design-for-discovery/ type: technical_blog published: 2023-06-15 reliability: high - id: src3 title: "Two Decades of Recommender Systems at Amazon.com" author: Brent Smith, Greg Linden url: https://assets.amazon.science/76/9e/7eac89c14a838746e91dde0a5e9f/two-decades-of-recommender-systems-at-amazon.pdf type: academic_paper published: 2017-05-01 reliability: authoritative - id: src4 title: "Recommendation Systems Overview" author: Google Developers url: https://developers.google.com/machine-learning/recommendation/overview/types type: official_docs published: 2024-01-15 reliability: authoritative - id: src5 title: "The Two-Tower Model for Recommendation Systems: A Deep Dive" author: Shaped AI url: https://www.shaped.ai/blog/the-two-tower-model-for-recommendation-systems-a-deep-dive type: technical_blog published: 2024-03-10 reliability: high - id: src6 title: "For Your Ears Only: Personalizing Spotify Home with Machine Learning" author: Spotify Engineering url: https://engineering.atspotify.com/2020/1/for-your-ears-only-personalizing-spotify-home-with-machine-learning type: technical_blog published: 2020-01-16 reliability: high - id: src7 title: "Approximate Nearest Neighbor Search in Recommender Systems" author: Zilliz (Milvus) url: https://zilliz.com/blog/approximate-nearest-neighbor-search-in-recommender-systems type: technical_blog published: 2024-02-20 reliability: moderate_high --- # How to Design a Recommendation Engine ## TL;DR - **Bottom line**: A production recommendation engine is a multi-stage pipeline — candidate retrieval (fast, coarse) followed by ranking (slow, precise) — backed by a feature store, an event stream, and an A/B testing framework. - **Key tool/command**: `Two-tower embedding model for retrieval + gradient-boosted or deep ranking model + FAISS/ScaNN ANN index` - **Watch out for**: Training-serving skew — features computed differently in training vs. serving silently degrade recommendation quality with no error signal. - **Works with**: Python (TensorFlow Recommenders, PyTorch, LightFM, Surprise), Spark, Kafka, Redis, FAISS, Feast, any cloud ML platform. ## Constraints - Never train on raw PII without anonymization — GDPR Article 22 and CCPA require consent for automated profiling and right-to-explanation - Always separate candidate retrieval from ranking — single-stage models cannot meet latency SLAs over catalogs >100K items - Feature store must guarantee training-serving consistency — use the same feature computation code for both paths - Never deploy a model without an A/B test against the production baseline — offline metric improvements (NDCG, recall@K) do not reliably predict online metric lifts (CTR, revenue) - Always implement a fallback for cold-start users/items — popularity-based or content-based defaults, never empty results ## Quick Reference | Component | Role | Technology Options | Scaling Strategy | |---|---|---|---| | Event Ingestion | Capture user interactions (clicks, views, purchases) in real-time | Kafka, AWS Kinesis, Google Pub/Sub | Partition by user_id; horizontal scaling | | Event Store | Durable log of all interaction events for replay and retraining | Kafka (retention), S3/GCS Parquet, Delta Lake | Tiered storage (hot/warm/cold) | | Feature Store | Serve consistent features for training and inference | Feast, Tecton, Vertex AI Feature Store | Online store (Redis/DynamoDB) + offline store (BigQuery/S3) | | Embedding Service | Generate user and item embeddings via two-tower model | TensorFlow Recommenders, PyTorch, custom | GPU cluster for training; CPU/GPU for inference | | ANN Index | Fast approximate nearest-neighbor retrieval over item embeddings | FAISS, ScaNN, Milvus, Pinecone, Weaviate | Shard by embedding space; replicate for read throughput | | Candidate Retrieval | Narrow millions of items to hundreds of candidates in <10ms | Two-tower model + ANN, co-occurrence, popularity | Multiple retrieval sources merged; each independently scalable | | Ranking Model | Score and re-rank candidates using rich cross-features | XGBoost, LambdaMART, deep ranking network (DCN, DLRM) | Model server (TF Serving, Triton); batch + online | | Business Rules Engine | Apply hard filters (age-gating, geo-restrictions, already-seen) | Custom service, Drools, OPA | Stateless; horizontal scaling | | Re-ranking / Diversity | Ensure result diversity, freshness, and business objectives | MMR, DPP, slot-based allocation | Lightweight post-processing; CPU-only | | A/B Testing Framework | Measure online impact of model changes on business KPIs | Optimizely, Statsig, custom (hashing-based) | Consistent hashing for user bucketing | | Model Training Pipeline | Retrain models on fresh data (daily or continuous) | Airflow/Kubeflow + Spark + GPU training | Scheduled DAGs; spot/preemptible instances | | Model Registry | Version, track, and deploy trained models | MLflow, Vertex AI Model Registry, SageMaker | Centralized; blue-green deployment | | Monitoring & Observability | Track model performance, data drift, and system health | Prometheus, Grafana, custom dashboards, Evidently AI | Alert on metric decay, feature drift, latency spikes | ## Decision Tree ``` START ├── Catalog size >100K items? │ ├── YES → Multi-stage pipeline required (retrieval + ranking) │ │ ├── Have rich item metadata (text, images, categories)? │ │ │ ├── YES → Hybrid: content-based retrieval + collaborative ranking │ │ │ └── NO → Pure collaborative filtering (two-tower + interaction data) │ │ └── Cold-start items common (>20% of catalog)? │ │ ├── YES → Content-based tower mandatory for item embedding │ │ └── NO → ID-based embeddings sufficient for item tower │ └── NO → Single-stage model feasible │ ├── Have explicit ratings (1-5 stars)? │ │ ├── YES → Matrix factorization (ALS/SVD) or LightFM │ │ └── NO → Implicit feedback model (BPR, WARP loss) │ └── Need real-time updates? │ ├── YES → Online learning or session-based model (GRU4Rec) │ └── NO → Batch retrained daily ├── Latency budget <50ms? │ ├── YES → Pre-compute recommendations; serve from cache/KV store │ └── NO → Online inference acceptable with feature store lookup └── DEFAULT → Start with popularity + simple collaborative filtering, iterate toward hybrid ``` ## Step-by-Step Guide ### 1. Define interaction schema and ingest events Design your event schema to capture every user-item interaction with context. This is the foundation — bad event data means bad recommendations regardless of model sophistication. [src1] ```python # Event schema (Avro/Protobuf recommended for production) interaction_event = { "user_id": "u_abc123", # anonymized user identifier "item_id": "item_789", # catalog item identifier "event_type": "click", # click | view | purchase | add_to_cart | skip "timestamp": "2026-02-23T10:30:00Z", "context": { "device": "mobile", "session_id": "sess_456", "page": "home_feed", "position": 3 # position in the list (for position bias correction) } } # Produce to Kafka topic # producer.send("user-interactions", key=user_id, value=interaction_event) ``` **Verify**: Check Kafka consumer lag stays <1s and event count matches expected traffic within 5%. ### 2. Build the feature store Materialize user features (historical interaction aggregates, demographics) and item features (metadata, popularity scores, freshness) into both offline and online stores. This eliminates training-serving skew. [src2] ```python # Feast feature definition (feast_repo/features.py) from feast import Entity, FeatureView, Field, FileSource from feast.types import Float32, Int64, String from datetime import timedelta user = Entity(name="user_id", join_keys=["user_id"]) user_features = FeatureView( name="user_features", entities=[user], ttl=timedelta(hours=24), schema=[ Field(name="total_clicks_7d", dtype=Int64), Field(name="avg_session_duration", dtype=Float32), Field(name="top_category", dtype=String), Field(name="interaction_count_30d", dtype=Int64), ], source=FileSource(path="s3://features/user_features.parquet"), ) # Materialize: feast materialize-incremental $(date -u +"%Y-%m-%dT%H:%M:%S") ``` **Verify**: `feast feature-server serve` responds in <5ms for user feature lookups; validate feature values match a manual SQL query on 10 random users. ### 3. Train the candidate retrieval model (two-tower) Train a two-tower model where the user tower and item tower produce embeddings in the same vector space. Similarity between embeddings determines relevance. [src4] [src5] ```python # Two-tower model with TensorFlow Recommenders import tensorflow as tf import tensorflow_recommenders as tfrs class TwoTowerModel(tfrs.Model): def __init__(self, user_model, item_model, task): super().__init__() self.user_model = user_model # user tower: user_id -> embedding self.item_model = item_model # item tower: item_id -> embedding self.task = task # retrieval task with in-batch negatives def compute_loss(self, features, training=False): user_embeddings = self.user_model(features["user_id"]) item_embeddings = self.item_model(features["item_id"]) return self.task(user_embeddings, item_embeddings) # Build towers user_model = tf.keras.Sequential([ tf.keras.layers.StringLookup(vocabulary=user_ids), tf.keras.layers.Embedding(len(user_ids) + 1, 64), tf.keras.layers.Dense(64, activation="relu"), tf.keras.layers.Dense(32), ]) item_model = tf.keras.Sequential([ tf.keras.layers.StringLookup(vocabulary=item_ids), tf.keras.layers.Embedding(len(item_ids) + 1, 64), tf.keras.layers.Dense(64, activation="relu"), tf.keras.layers.Dense(32), ]) task = tfrs.tasks.Retrieval( metrics=tfrs.metrics.FactorizedTopK( candidates=items_dataset.batch(128).map(item_model) ) ) model = TwoTowerModel(user_model, item_model, task) model.compile(optimizer=tf.keras.optimizers.Adagrad(0.1)) model.fit(train_dataset.batch(4096), epochs=5) ``` **Verify**: Evaluate recall@100 on held-out test set; target >0.25 for initial model. Check that embedding dimensions match between towers. ### 4. Build the ANN index for fast retrieval Export item embeddings and build an approximate nearest-neighbor index for sub-millisecond retrieval of top-K candidates from the full catalog. [src7] ```python import faiss import numpy as np # Extract all item embeddings (shape: [num_items, embedding_dim]) item_embeddings_np = np.array([ item_model(tf.constant([item_id])).numpy()[0] for item_id in item_ids ], dtype="float32") # Build IVF index for large catalogs embedding_dim = 32 nlist = 256 # number of clusters (tune: sqrt(num_items) is a starting point) quantizer = faiss.IndexFlatIP(embedding_dim) # inner product similarity index = faiss.IndexIVFFlat(quantizer, embedding_dim, nlist, faiss.METRIC_INNER_PRODUCT) index.train(item_embeddings_np) index.add(item_embeddings_np) index.nprobe = 16 # search 16 clusters (trade-off: speed vs recall) # Query: get top 200 candidates for a user user_embedding = user_model(tf.constant(["u_abc123"])).numpy() scores, indices = index.search(user_embedding, k=200) candidate_item_ids = [item_ids[i] for i in indices[0]] ``` **Verify**: Measure recall@200 vs. brute-force search; target >0.95. Measure query latency; target <5ms for 1M items. ### 5. Build the ranking model Train a ranking model that takes the retrieved candidates and produces a fine-grained relevance score using rich cross-features between user and item. [src2] [src3] ```python # Ranking model with cross-features (simplified) import xgboost as xgb import pandas as pd # Feature engineering for each (user, candidate_item) pair def build_ranking_features(user_id, candidate_items, feature_store): user_feats = feature_store.get_online_features( entity_rows=[{"user_id": user_id}], features=["user_features:total_clicks_7d", "user_features:top_category"] ).to_dict() rows = [] for item_id in candidate_items: item_feats = feature_store.get_online_features( entity_rows=[{"item_id": item_id}], features=["item_features:category", "item_features:popularity_score", "item_features:days_since_publish"] ).to_dict() rows.append({ "user_clicks_7d": user_feats["total_clicks_7d"][0], "category_match": int(user_feats["top_category"][0] == item_feats["category"][0]), "item_popularity": item_feats["popularity_score"][0], "item_freshness": item_feats["days_since_publish"][0], }) return pd.DataFrame(rows) # Train with pairwise ranking objective dtrain = xgb.DMatrix(train_features, label=train_labels) params = { "objective": "rank:pairwise", "eval_metric": "ndcg@10", "max_depth": 6, "eta": 0.1, } ranker = xgb.train(params, dtrain, num_boost_round=200) ``` **Verify**: Evaluate NDCG@10 on held-out set; target >0.35. Compare against popularity-only baseline to confirm model adds value. ### 6. Apply business rules and diversity re-ranking After scoring, apply hard business rules (geo-restrictions, already-consumed filtering) and diversity logic to avoid filter bubbles. [src6] ```python def apply_business_rules_and_diversity(ranked_items, user_history, rules): # Step 1: Hard filters filtered = [item for item in ranked_items if item["id"] not in user_history and item["region"] in rules["allowed_regions"] and item["age_rating"] <= rules["user_age_rating"]] # Step 2: Maximal Marginal Relevance (MMR) for diversity selected = [] lambda_param = 0.7 # 0=max diversity, 1=max relevance remaining = filtered.copy() while len(selected) < rules["num_results"] and remaining: best_score = -float("inf") best_idx = 0 for i, candidate in enumerate(remaining): relevance = candidate["score"] max_sim = max( (cosine_similarity(candidate["embedding"], s["embedding"]) for s in selected), default=0 ) mmr = lambda_param * relevance - (1 - lambda_param) * max_sim if mmr > best_score: best_score = mmr best_idx = i selected.append(remaining.pop(best_idx)) return selected ``` **Verify**: Check that final result set has no duplicate categories in top-3 positions and no already-consumed items appear. ### 7. Deploy with A/B testing and monitoring Serve the pipeline behind an A/B testing framework. Monitor model metrics (NDCG, coverage, diversity), system metrics (latency, throughput), and business metrics (CTR, conversion, revenue). [src1] ```yaml # A/B test configuration experiment: name: "rec_v2_two_tower" allocation: control: 50% # existing model treatment: 50% # new two-tower pipeline primary_metric: revenue_per_user guardrail_metrics: - p99_latency_ms: <200 - recommendation_coverage: >0.60 - category_diversity: >0.40 duration_days: 14 min_sample_size: 50000 ``` **Verify**: Confirm consistent user bucketing (same user always sees same variant). Check that metric tracking fires correctly by inspecting 100 random events in the analytics pipeline. ## Code Examples ### Python: Collaborative Filtering with Implicit Feedback (Surprise Library) ```python # Input: CSV of (user_id, item_id, rating) tuples # Output: Top-N recommendations for a given user from surprise import SVD, Dataset, Reader from surprise.model_selection import cross_validate import pandas as pd reader = Reader(rating_scale=(1, 5)) data = Dataset.load_from_df( pd.read_csv("ratings.csv")[["user_id", "item_id", "rating"]], reader ) algo = SVD(n_factors=100, n_epochs=20, lr_all=0.005, reg_all=0.02) cross_validate(algo, data, measures=["RMSE", "MAE"], cv=5, verbose=True) trainset = data.build_full_trainset() algo.fit(trainset) # Get top-10 for user "u_123" anti_testset = trainset.build_anti_testset() predictions = algo.test([x for x in anti_testset if x[0] == "u_123"]) top_10 = sorted(predictions, key=lambda x: x.est, reverse=True)[:10] print([(p.iid, round(p.est, 2)) for p in top_10]) ``` ### Python: Feature Store Pattern with Feast ```python # Input: user_id at serving time # Output: consistent feature vector for model inference from feast import FeatureStore store = FeatureStore(repo_path="feast_repo/") # Online serving — same features used in training features = store.get_online_features( entity_rows=[{"user_id": "u_abc123"}], features=[ "user_features:total_clicks_7d", "user_features:avg_session_duration", "user_features:top_category", "item_features:popularity_score", ], ).to_dict() # Pass features to ranking model for inference # This guarantees no training-serving skew ``` ## Anti-Patterns ### Wrong: Training on all interactions equally ```python # BAD — treating all events as equal positive signals train_data = df[["user_id", "item_id"]] # clicks, views, purchases all weight=1 model.fit(train_data) ``` ### Correct: Weight interactions by signal strength ```python # GOOD — stronger signals (purchase > add_to_cart > click > view) get higher weight weights = {"purchase": 5.0, "add_to_cart": 3.0, "click": 1.0, "view": 0.3} df["weight"] = df["event_type"].map(weights) model.fit(df[["user_id", "item_id"]], sample_weight=df["weight"]) ``` ### Wrong: Using the same features for retrieval and ranking ```python # BAD — retrieval model uses cross-features (slow, cannot scale to full catalog) retrieval_features = ["user_age", "item_category", "user_x_item_category_affinity"] # This requires computing features for ALL items per request — O(N) is too slow ``` ### Correct: Retrieval uses independent towers, ranking uses cross-features ```python # GOOD — retrieval uses independent user/item towers (pre-computed, ANN lookup) # Ranking model uses rich cross-features but only on ~200 candidates user_embedding = user_tower(user_features) # retrieval: independent item_embeddings = ann_index.search(user_embedding, k=200) # O(log N) ranked = ranking_model.predict(cross_features(user, candidates)) # O(200) ``` ### Wrong: Evaluating only with offline metrics ```python # BAD — deploying based solely on NDCG improvement if new_model_ndcg > old_model_ndcg: deploy(new_model) # no A/B test, no business metric validation ``` ### Correct: A/B test with business metrics as primary KPI ```python # GOOD — offline metrics gate A/B test launch; online metrics gate full rollout if new_model_ndcg > old_model_ndcg * 1.02: # >2% offline lift launch_ab_test(new_model, traffic=10%) # Only promote to 100% if revenue_per_user improves with p<0.05 ``` ### Wrong: No position bias correction ```python # BAD — items shown in position 1 get more clicks regardless of relevance # Training on this data reinforces existing ranking, creating a feedback loop train_data = raw_click_log # position 1 has 10x CTR of position 10 ``` ### Correct: Apply inverse propensity weighting or position-aware training ```python # GOOD — correct for position bias in training data position_bias = {1: 1.0, 2: 0.7, 3: 0.5, 4: 0.35, 5: 0.25} # learned from randomization df["corrected_weight"] = df["clicked"] / df["position"].map(position_bias) # Or use a position feature in the model and zero it out at serving time ``` ## Common Pitfalls - **Training-serving skew**: Features computed differently in batch training vs. online serving cause silent quality degradation. Fix: Use a feature store (Feast/Tecton) that serves the exact same feature computation for both paths. [src2] - **Popularity bias amplification**: Models trained on click data over-recommend popular items, creating a filter bubble. Fix: Add exploration (epsilon-greedy or Thompson sampling) and diversity re-ranking (MMR with lambda=0.7). [src6] - **Cold-start item neglect**: New items receive zero impressions because they have no interaction data. Fix: Use content-based features (title embeddings, category, metadata) in the item tower so new items get meaningful embeddings from day one. [src4] - **Ignoring position bias**: Items at top positions get disproportionate clicks, creating a feedback loop that reinforces the current ranking. Fix: Apply inverse propensity scoring or include position as a training feature (set to zero at inference). [src2] - **Stale embeddings**: Item embeddings computed once and never refreshed mean new interactions and catalog changes are ignored. Fix: Rebuild ANN index at least daily; retrain embeddings weekly or continuously. [src7] - **Over-engineering for small catalogs**: Building a multi-stage pipeline for <10K items wastes engineering effort. Fix: Use matrix factorization (SVD/ALS) or LightFM — a single model with brute-force scoring is fast enough. [src3] - **Neglecting implicit negative signals**: Treating only clicks as positives ignores skips, scroll-pasts, and short view durations. Fix: Use weighted implicit feedback (view <5s = negative, view >30s = weak positive, click = positive, purchase = strong positive). [src1] - **No fallback for empty results**: When retrieval returns zero candidates (new user, niche query), the system returns an empty page. Fix: Always merge retrieval sources (personalized + popularity + trending) so at least one produces results. [src6] ## When to Use / When Not to Use | Use When | Don't Use When | Use Instead | |---|---|---| | Catalog >1K items and user interaction data available | Catalog <100 items | Manual curation or simple rule-based sorting | | Personalization drives business KPI (engagement, revenue) | All users should see the same content (editorial, news homepage) | Content management system with editorial ranking | | Sufficient interaction volume (>10K interactions/day) | Very sparse data (<1K total interactions) | Content-based filtering only, or popularity-based | | Need to surface long-tail items users would not find via search | Users know exactly what they want (transactional search) | Search engine with relevance ranking | | Real-time personalization matters (feeds, homepages) | Batch recommendations suffice (weekly email digest) | Simpler batch job with matrix factorization | ## Important Caveats - Offline metric improvements (NDCG, recall@K) frequently do not translate to online business metric gains — always A/B test before full rollout - GDPR Article 22 grants users the right to explanation for automated decisions including recommendations — design for explainability from day one (e.g., "Because you watched X") - Recommendation quality degrades rapidly when interaction data is stale — budget for daily retraining pipelines and real-time feature updates - Two-tower models trade off cross-feature expressiveness for retrieval speed — the ranking stage must compensate with richer feature interactions - Position bias in training data creates a rich-get-richer feedback loop — invest in randomized exploration early to collect unbiased signal ## Related Units - [Feature Store Architecture](/software/system-design/feature-store/2026) - [Search Engine Design](/software/system-design/search-engine/2026) - [Ad Ranking System Design](/software/system-design/ad-ranking-system/2026) - [Real-Time Analytics Pipeline](/software/system-design/real-time-analytics/2026) - [Rule-Based Recommendation System](/software/system-design/rule-based-recommendations/2026)