---
# === IDENTITY ===
id: business/startup-metrics/experiment-tracking-framework/2026
canonical_question: "How do I track experiments — hypothesis, test, measure, decide methodology, minimum sample sizes, significance?"
aliases:
  - "Experiment tracking framework for startups 2026"
  - "A/B testing methodology and sample size guide"
  - "Hypothesis-driven experiment management system"
entity_type: execution_recipe
domain: business > startup-metrics > experiment-tracking-framework
region: global
jurisdiction: global
temporal_scope: 2025-2026

# === VERIFICATION ===
last_verified: 2026-03-12
confidence: 0.88
version: 1.0
first_published: 2026-03-12

# === TEMPORAL VALIDITY ===
temporal_validity:
  status: evolving
  last_breaking_change: "Eppo and Statsig introduced auto-experiments and customizable significance thresholds in 2025, moving beyond rigid p < 0.05 defaults"
  next_review: 2026-09-08
  change_sensitivity: high

# === CONSTRAINTS ===
constraints:
  - "Sample size must be calculated BEFORE launching any test — never peek at results and stop early without sequential testing correction"
  - "Statistical significance threshold (alpha) must be declared upfront — post-hoc threshold adjustment invalidates results"
  - "Minimum detectable effect (MDE) must be realistic for the product's traffic — overly ambitious MDEs produce underpowered tests"
  - "Run experiments for full business cycles (minimum 1-2 weeks) to capture day-of-week and pay-cycle effects"
  - "Multiple metric testing requires Bonferroni or false discovery rate correction — testing 10 metrics without correction inflates false positives to ~40%"

# === SKIP CONDITIONS ===
skip_this_unit_if:
  - condition: "User needs a strategic experimentation program design, not execution"
    use_instead: "business/startup-metrics/experimentation-program-design"
  - condition: "User needs product analytics setup, not experiment tracking"
    use_instead: "business/startup-metrics/product-analytics-setup"
  - condition: "User runs fewer than 100 visitors/week — experiments require statistical power"
    use_instead: "business/startup-metrics/qualitative-testing-guide"

# === AGENT HINTS ===
inputs_needed:
  - key: tool_preference
    question: "Which experimentation platform should be used?"
    type: choice
    options: ["Optimizely", "VWO", "Statsig", "Eppo", "PostHog", "Google Optimize successor (custom)", "spreadsheet-only", "no preference — auto-select"]
  - key: technical_skill
    question: "What is the user's technical skill level?"
    type: choice
    options: ["non-technical (no-code only)", "semi-technical (can edit code)", "developer (can write code)"]
  - key: budget_for_tools
    question: "What's the tool/API budget?"
    type: choice
    options: ["free tier only", "up to $50/month", "up to $200/month", "no limit"]
  - key: scale
    question: "What traffic volume does the product receive?"
    type: choice
    options: ["low (100-1K visitors/week)", "medium (1K-10K visitors/week)", "high (10K-100K visitors/week)", "very high (100K+ visitors/week)"]

# === EXECUTION METADATA ===
execution:
  required_inputs:
    - name: "Product metrics baseline"
      source: "analytics platform or business/startup-metrics/metric-definition-framework"
      format: "structured data — current conversion rates, averages, standard deviations"
    - name: "Experiment hypothesis"
      source: "product team or business/startup-metrics/prioritization-framework"
      format: "document — observation, proposed change, expected outcome"

  outputs:
    - name: "Experiment tracker spreadsheet/database"
      format: "spreadsheet or platform dashboard"
      description: "Central log of all experiments with hypothesis, status, sample size, results, and decision"
    - name: "Individual experiment report"
      format: "document"
      description: "Per-experiment analysis with statistical results, confidence intervals, and go/no-go decision"
    - name: "Learnings repository"
      format: "document or wiki"
      description: "Accumulated experiment insights indexed by feature area, metric, and outcome"

  tools_required:
    - name: "Evan Miller Sample Size Calculator"
      purpose: "Pre-test sample size calculation"
      tier: free
      cost: "$0"
      alternatives: ["Statsig power analysis tool", "Eppo sample size calculator", "custom R/Python script"]
    - name: "Experimentation platform"
      purpose: "Experiment assignment, tracking, and statistical analysis"
      tier: "free-to-paid"
      cost: "$0 (PostHog free) to $500+/mo (Optimizely)"
      alternatives: ["Statsig", "Eppo", "VWO", "PostHog", "LaunchDarkly", "custom build"]
    - name: "Experiment tracker"
      purpose: "Central experiment log and hypothesis management"
      tier: free
      cost: "$0"
      alternatives: ["Google Sheets", "Notion", "Optimizely Collaboration", "VWO Plan"]

  credentials_needed:
    - service: "Experimentation platform"
      type: "API key or SDK token"
      where_to_get: "Platform dashboard (Statsig: statsig.com, Eppo: geteppo.com, PostHog: posthog.com)"
      free_tier_limits: "PostHog: 1M events/mo free; Statsig: 10M events/mo free"

  estimated_duration: "2-4 hours initial setup; 15-30 minutes per experiment launch; 1-4 weeks per experiment run"
  estimated_cost: "$0 (free tier + spreadsheet) to $500+/month (enterprise platform)"

# === DISTRIBUTION ===
canonical_source: "https://knowledgelib.io/business/startup-metrics/experiment-tracking-framework/2026"
suggested_citation: "Source: knowledgelib.io — AI Knowledge Library (verified 2026-03-12)"

# === RELATED UNITS ===
related_kos:
  depends_on:
    - id: "business/startup-metrics/metric-definition-framework"
      label: "Metric definitions needed before designing experiments"
  feeds_into:
    - id: "business/startup-metrics/growth-model-calibration"
      label: "Experiment results feed growth model updates"
    - id: "business/startup-dashboard/experiment-dashboard"
      label: "Dashboard displaying experiment portfolio status"
  related_to:
    - id: "business/startup-metrics/north-star-metric-selection"
      label: "North star metric determines primary experiment OEC"
  alternative_to:
    - id: "business/startup-metrics/qualitative-testing-guide"
      label: "For low-traffic products where statistical testing is infeasible"

# === SOURCES ===
sources:
  - id: src1
    title: "Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing"
    author: "Ron Kohavi, Diane Tang, Ya Xu"
    url: https://experimentguide.com/
    type: academic_book
    published: 2020-05-01
    reliability: authoritative
  - id: src2
    title: "Sample Size Calculator"
    author: "Evan Miller"
    url: https://www.evanmiller.org/ab-testing/sample-size.html
    type: tool
    published: 2024-01-15
    reliability: authoritative
  - id: src3
    title: "How to do a power analysis to determine sample size"
    author: Statsig
    url: https://www.statsig.com/perspectives/power-analysis-determine-sample-size
    type: technical_guide
    published: 2025-01-10
    reliability: authoritative
  - id: src4
    title: "Experimentation Frameworks: A Full Guide With Examples"
    author: Eppo
    url: https://www.geteppo.com/blog/what-is-an-experimentation-framework
    type: technical_guide
    published: 2025-03-01
    reliability: authoritative
  - id: src5
    title: "The Optimization Methodology"
    author: Optimizely
    url: https://support.optimizely.com/hc/en-us/articles/4410283328013-The-Optimization-Methodology
    type: official_docs
    published: 2025-01-15
    reliability: authoritative
  - id: src6
    title: "A/B Testing Template: Plan, Run & Optimize Tests"
    author: VWO
    url: https://vwo.com/blog/ab-testing-template/
    type: technical_guide
    published: 2025-02-01
    reliability: authoritative
---

# Experiment Tracking Framework

## Purpose

This recipe produces a complete experiment tracking system — from structured hypothesis documentation through sample size calculation, test execution, statistical analysis, and decision logging. The output is a reusable framework that ensures every product experiment follows rigorous methodology: declared hypotheses, pre-calculated sample sizes, significance thresholds, and documented decisions with learnings. Prevents the most common startup failure mode where teams "run experiments" by shipping changes and checking metrics a few days later without statistical rigor. [src1]

## Prerequisites
<!-- Agents: verify ALL prerequisites before executing. Missing prerequisites = failed execution. -->

- [ ] **Baseline metrics** available — current conversion rates (or relevant KPIs) with at least 2-4 weeks of historical data
- [ ] **Traffic estimate** — weekly unique visitors or events for the metric being tested
- [ ] **Experimentation platform account** — [Statsig](https://statsig.com) (free: 10M events/mo), [PostHog](https://posthog.com) (free: 1M events/mo), or [VWO](https://vwo.com) (free trial)
- [ ] **Experiment tracker** — Google Sheets, Notion database, or platform-native hypothesis dashboard ([Optimizely Collaboration](https://www.optimizely.com/products/feature-experimentation/), [VWO Plan](https://vwo.com/plan/))
- [ ] **Stakeholder alignment** — agreement on primary metric (Overall Evaluation Criterion) and guardrail metrics

## Constraints
<!-- Hard rules. Agents: enforce throughout execution. Violating these = broken output or legal risk. -->

- Never peek at results and stop early without sequential testing correction — this inflates false positive rates to 25-30%. [src1]
- Declare statistical significance threshold (typically alpha = 0.05) and minimum detectable effect BEFORE launching. Post-hoc adjustment is p-hacking. [src1]
- Run experiments for at least 1-2 full business cycles (7-14 days minimum) regardless of sample size achievement, to capture weekly seasonality. [src1]
- Testing multiple metrics simultaneously requires correction — Bonferroni (conservative) or Benjamini-Hochberg FDR (practical). Without correction, 10 metrics at alpha 0.05 yields ~40% false positive probability.
- Never run overlapping experiments on the same user population affecting the same metric without interaction analysis — confounding makes both results unreliable.

## Tool Selection Decision

<!-- Agent selects the right tool path based on user inputs.
     Each path leads to a different execution flow below. -->

```
Which path?
├── User is non-technical AND budget = free
│   └── PATH A: No-Code Free — Google Sheets tracker + Evan Miller calculator + free PostHog
├── User is non-technical AND budget > $0
│   └── PATH B: No-Code Paid — VWO or Optimizely (built-in hypothesis management + visual editor)
├── User is semi-technical or developer AND budget = free
│   └── PATH C: Code + Free — Statsig/PostHog SDK + spreadsheet tracker + Evan Miller calculator
└── User is developer AND budget > $0
    └── PATH D: Code + Paid — Statsig/Eppo + warehouse-native analysis + automated power analysis
```

| Path | Tools | Cost | Speed | Output Quality |
|------|-------|------|-------|---------------|
| A: No-Code Free | Google Sheets + Evan Miller + PostHog | $0 | Manual, 30 min/experiment setup | Good — rigorous if discipline maintained |
| B: No-Code Paid | VWO Plan or Optimizely | $200-500/mo | Fast, 10-15 min/experiment setup | Excellent — enforced workflow |
| C: Code + Free | Statsig/PostHog SDK + Sheets | $0 | 20 min/experiment setup | Very Good — SDK automates assignment |
| D: Code + Paid | Statsig/Eppo + warehouse | $200-1000/mo | Fastest, 10 min/experiment setup | Excellent — automated power analysis and CUPED |

## Execution Flow

### Step 1: Write the Hypothesis

**Duration**: 10-20 minutes
**Tool**: Experiment tracker (spreadsheet or platform)

Write a structured hypothesis for every experiment before any implementation. Use the format: "If [specific change], then [measurable outcome] will [increase/decrease] by [estimated magnitude], because [reasoning based on data/observation]." [src4]

Every hypothesis entry must include:

```
Experiment ID:     EXP-{YYYY}-{NNN}
Date proposed:     {YYYY-MM-DD}
Hypothesis:        If [change], then [metric] will [direction] by [magnitude], because [reasoning]
Primary metric:    {Overall Evaluation Criterion — e.g., signup conversion rate}
Guardrail metrics: {Metrics that must NOT degrade — e.g., page load time, revenue per user}
Owner:             {Name}
Status:            Proposed → Designed → Running → Analyzing → Decided
Decision:          {Ship | Iterate | Kill | Inconclusive}
```

**Verify**: Hypothesis includes a specific, falsifiable prediction with a named metric and estimated effect size.
**If failed**: If the team cannot state a specific expected outcome, the experiment is exploratory — document it as such and define what "interesting" means before launching.

### Step 2: Calculate Sample Size

**Duration**: 5-10 minutes
**Tool**: [Evan Miller Sample Size Calculator](https://www.evanmiller.org/ab-testing/sample-size.html) or Statsig Power Analysis [src2] [src3]

Calculate the minimum sample size per variation BEFORE launching. Required inputs:

- **Baseline conversion rate**: Current metric value (e.g., 3.2% signup rate)
- **Minimum Detectable Effect (MDE)**: Smallest improvement worth detecting (e.g., 0.5 percentage points, meaning 3.2% to 3.7%)
- **Statistical significance level (alpha)**: Typically 0.05 (5% false positive rate)
- **Statistical power (1 - beta)**: Typically 0.80 (80% chance of detecting a real effect)

```
Sample Size Formula (two-tailed, two-sample):
n = (Z_{alpha/2} + Z_{beta})^2 * (p1*(1-p1) + p2*(1-p2)) / (p2 - p1)^2

Where:
  Z_{alpha/2} = 1.96  (for alpha = 0.05, two-tailed)
  Z_{beta}    = 0.84  (for power = 0.80)
  p1          = baseline conversion rate
  p2          = baseline + MDE

Example: baseline = 3.2%, MDE = 0.5pp (target = 3.7%)
  n = (1.96 + 0.84)^2 * (0.032*0.968 + 0.037*0.963) / (0.005)^2
  n ≈ 17,500 per variation (35,000 total for A/B)
```

Then estimate runtime:

```
Runtime = (sample_size_per_variation * number_of_variations) / weekly_traffic * 7 days

Example: 35,000 total / 5,000 visitors per week = 7 weeks
         If this exceeds acceptable duration, increase MDE or accept lower power.
```

**Verify**: Sample size is documented in the experiment tracker. Runtime fits within business constraints (typically 2-8 weeks maximum).
**If failed**: If required runtime exceeds 8 weeks — either increase MDE (detect only larger effects), increase traffic allocation, or use variance reduction (CUPED) to reduce required sample size by 20-50%. [src3]

### Step 3: Design the Experiment

**Duration**: 15-30 minutes
**Tool**: Experimentation platform + tracker

Document the full experiment design before building anything:

```
EXPERIMENT DESIGN DOCUMENT
─────────────────────────
Experiment ID:       EXP-2026-042
Hypothesis:          If we simplify the signup form from 5 fields to 3 fields,
                     then signup conversion will increase by >= 0.5pp (3.2% → 3.7%),
                     because analytics shows 47% drop-off on the 4th field.

Primary metric (OEC): Signup completion rate
Guardrail metrics:    - Account quality score (must not drop > 5%)
                      - 7-day retention (must not drop > 2%)
                      - Revenue per user (must not drop > 3%)

Traffic allocation:   50% control / 50% treatment
Sample size:          17,500 per variation (35,000 total)
Estimated runtime:    7 weeks at current traffic
Significance level:   alpha = 0.05 (two-tailed)
Power:                0.80
Analysis method:      Fixed-sample (analyze once at full sample size)

Randomization unit:   User ID (cookie-based for anonymous, account-based for logged-in)
Targeting:            All users on /signup page, desktop + mobile
Exclusions:           Bot traffic, internal IPs, users in other active signup experiments
```

**Verify**: Design document reviewed by at least one other person. Guardrail metrics defined. No overlap with running experiments on same population.
**If failed**: If overlapping experiments exist, either pause one or implement mutual exclusion groups in the experimentation platform.

### Step 4: Implement Tracking

**Duration**: 30-60 minutes (code path) or 15-30 minutes (no-code path)
**Tool**: Experimentation platform SDK or visual editor

For code-based implementation (Paths C/D):

```javascript
// Statsig example — server-side assignment
const statsig = require('statsig-node');
await statsig.initialize('server-secret-key');

// Get experiment assignment for user
const experiment = statsig.getExperiment(user, 'exp_signup_simplification');
const variant = experiment.get('variant', 'control'); // 'control' or 'treatment'

// Track exposure (automatic in most SDKs)
// Track conversion event
statsig.logEvent(user, 'signup_completed', null, {
  experiment_id: 'exp_signup_simplification',
  variant: variant
});
```

For no-code implementation (Paths A/B):

```
VWO / Optimizely workflow:
1. Create experiment in platform dashboard
2. Use visual editor to make the change (e.g., hide form fields)
3. Set targeting rules (URL, audience, device)
4. Configure goals: primary metric + guardrails
5. Set traffic allocation (50/50 for A/B)
6. QA in preview mode — verify both variations render correctly
7. Launch experiment
```

**Verify**: Run QA checks — both control and treatment display correctly. Events fire on expected user actions. Check experiment platform shows incoming data within first 30 minutes.
**If failed**: If no data appears, check SDK initialization, event naming, and ad-blocker interference. Use platform debug mode.

### Step 5: Run Test to Statistical Significance

**Duration**: 1-8 weeks depending on traffic and MDE
**Tool**: Experimentation platform dashboard

Monitor the experiment without making decisions until the pre-calculated sample size is reached:

```
MONITORING CHECKLIST (check weekly):
─────────────────────────────────
□ Sample ratio mismatch (SRM) check:
  - Control and treatment sample sizes within 1% of expected ratio
  - If SRM detected → STOP experiment, investigate randomization bug

□ Data quality check:
  - Conversion rates are in expected range (not 0% or 100%)
  - No logging gaps or duplicate events
  - Bot traffic filtered

□ Guardrail check:
  - No guardrail metric has degraded beyond the pre-set threshold
  - If critical guardrail violated → consider early stopping

□ DO NOT:
  - Stop the experiment because you see a "winning" variant early
  - Extend the experiment because results aren't significant yet
  - Change targeting or traffic allocation mid-experiment
  - Add or remove metrics mid-experiment
```

If using sequential testing (Statsig, Eppo support this): the platform adjusts confidence intervals continuously and you can make valid decisions at any point the platform declares significance. [src3]

**Verify**: Experiment reached pre-calculated sample size. No SRM detected. Guardrails intact.
**If failed**: If SRM > 1% — stop experiment, investigate and fix randomization issue, restart with new experiment ID. Do not reuse contaminated data.

### Step 6: Analyze Results

**Duration**: 30-60 minutes
**Tool**: Experimentation platform + spreadsheet

Once the sample size target is met, analyze results:

```
ANALYSIS TEMPLATE
─────────────────
Experiment:        EXP-2026-042
Runtime:           7 weeks (2026-01-15 to 2026-03-05)
Sample size:       Control: 17,823 | Treatment: 17,691

PRIMARY METRIC: Signup conversion rate
  Control:    3.21% (573/17,823)
  Treatment:  3.74% (662/17,691)
  Lift:       +0.53pp (+16.5% relative)
  95% CI:     [+0.12pp, +0.94pp]
  p-value:    0.011
  Significant: YES (p < 0.05)

GUARDRAIL METRICS:
  Account quality:  Control 72.1 vs Treatment 71.8 → -0.4% (within -5% threshold) ✓
  7-day retention:  Control 34.2% vs Treatment 33.9% → -0.3pp (within -2pp threshold) ✓
  Revenue/user:     Control $4.21 vs Treatment $4.18 → -0.7% (within -3% threshold) ✓

SEGMENTS (exploratory — not primary decision criteria):
  Mobile:   +0.71pp lift (p=0.03)
  Desktop:  +0.38pp lift (p=0.12)
  New users: +0.62pp lift (p=0.008)
```

**Verify**: Confidence interval does not cross zero for a significant result. Guardrail metrics within acceptable bounds. Segment analysis flagged as exploratory (not used for primary go/no-go decision).
**If failed**: If results are inconclusive (CI crosses zero) — document as inconclusive. Do NOT extend the experiment hoping for significance. Either accept the null result or redesign with larger MDE.

### Step 7: Make Decision

**Duration**: 15-30 minutes
**Tool**: Experiment tracker

Apply a structured decision framework:

```
DECISION MATRIX
───────────────
Primary metric significant + positive AND guardrails intact
  → SHIP the treatment

Primary metric significant + positive BUT guardrail violated
  → ITERATE — redesign to preserve guardrail metric

Primary metric NOT significant AND estimated lift within MDE
  → INCONCLUSIVE — the experiment was too small to detect the effect
  → Option: re-run with higher power or accept null result

Primary metric significant + NEGATIVE
  → KILL — revert to control, document the anti-pattern

Primary metric NOT significant AND estimated lift near zero
  → NULL RESULT — the change has no meaningful effect
  → Kill or ship based on engineering simplicity (no metric impact either way)
```

**Verify**: Decision recorded in experiment tracker with rationale. Stakeholders notified.
**If failed**: If team disagrees on decision despite clear data — escalate to pre-agreed decision-maker. Data breaks ties, not opinions (avoid HiPPO — Highest Paid Person's Opinion). [src1]

### Step 8: Document Learnings

**Duration**: 15-20 minutes
**Tool**: Experiment tracker + learnings repository

Create a permanent record that compounds institutional knowledge:

```
EXPERIMENT CONCLUSION
─────────────────────
Experiment:    EXP-2026-042 — Simplified Signup Form
Decision:      SHIP treatment
Outcome:       +0.53pp signup conversion (+16.5% relative)
Projected annual impact: +2,760 additional signups → ~$138K incremental revenue

KEY LEARNINGS:
1. Reducing form fields from 5 to 3 increased completion by 16.5%
2. Removed fields (company size, role) had no impact on account quality
3. Mobile users benefited most — mobile signup was the primary bottleneck
4. Team initially resisted because "we need that data" — experiment proved
   the data wasn't worth the conversion cost

FOLLOW-UP EXPERIMENTS:
- Test progressive profiling (collect removed fields post-signup)
- Test single-field signup (email only + magic link)

TAGS: #signup #form-optimization #mobile #conversion #ship
```

**Verify**: Learnings searchable by tag, feature area, and outcome. Follow-up experiments added to backlog.
**If failed**: If learnings are not being consulted before new experiments — institute a mandatory "prior art search" step in hypothesis creation.

## Output Schema

<!-- Exact format of the deliverable. Downstream agents and dashboard cards
     reference this schema to consume the output. -->

```json
{
  "output_type": "experiment_tracker",
  "format": "spreadsheet or database",
  "columns": [
    {"name": "experiment_id", "type": "string", "description": "Unique ID (EXP-YYYY-NNN)", "required": true},
    {"name": "date_proposed", "type": "date", "description": "When hypothesis was written", "required": true},
    {"name": "hypothesis", "type": "string", "description": "If/then/because statement", "required": true},
    {"name": "primary_metric", "type": "string", "description": "Overall Evaluation Criterion", "required": true},
    {"name": "guardrail_metrics", "type": "string[]", "description": "Metrics that must not degrade", "required": true},
    {"name": "baseline_rate", "type": "number", "description": "Current metric value", "required": true},
    {"name": "mde", "type": "number", "description": "Minimum detectable effect", "required": true},
    {"name": "sample_size_per_variation", "type": "number", "description": "Required n per variation", "required": true},
    {"name": "estimated_runtime_weeks", "type": "number", "description": "Weeks to reach sample size", "required": true},
    {"name": "status", "type": "string", "description": "Proposed|Designed|Running|Analyzing|Decided", "required": true},
    {"name": "start_date", "type": "date", "description": "When experiment launched", "required": false},
    {"name": "end_date", "type": "date", "description": "When experiment concluded", "required": false},
    {"name": "result_lift", "type": "number", "description": "Observed lift (absolute)", "required": false},
    {"name": "result_ci_lower", "type": "number", "description": "95% CI lower bound", "required": false},
    {"name": "result_ci_upper", "type": "number", "description": "95% CI upper bound", "required": false},
    {"name": "result_p_value", "type": "number", "description": "p-value", "required": false},
    {"name": "decision", "type": "string", "description": "Ship|Iterate|Kill|Inconclusive", "required": false},
    {"name": "learnings", "type": "string", "description": "Key takeaways", "required": false},
    {"name": "tags", "type": "string[]", "description": "Searchable tags", "required": false}
  ],
  "expected_row_count": "10-200 per quarter depending on team velocity",
  "sort_order": "date_proposed descending",
  "deduplication_key": "experiment_id"
}
```

## Quality Benchmarks

<!-- How to evaluate if the output is good enough.
     Links to benchmark cards where applicable. -->

| Quality Metric | Minimum Acceptable | Good | Excellent |
|---------------|-------------------|------|-----------|
| Hypothesis completion rate (% with full if/then/because) | > 70% | > 85% | > 95% |
| Pre-calculated sample size rate (% of experiments) | > 80% | > 95% | 100% |
| Experiment win rate (% of experiments that ship) | > 15% | > 25% | > 35% |
| Sample ratio mismatch detection rate | > 90% | > 95% | 100% |
| Learnings documented rate (% of concluded experiments) | > 60% | > 80% | > 95% |
| Average experiment velocity (experiments concluded/quarter) | > 3 | > 8 | > 15 |

**If below minimum**: If hypothesis completion is low, enforce the structured template — reject experiments without full documentation. If win rate is below 15%, improve hypothesis quality by requiring data-backed observations before proposing experiments.

## Error Handling

<!-- Common failures and recovery actions. Agents need this to handle
     execution errors without human intervention. -->

| Error | Likely Cause | Recovery Action |
|-------|-------------|----------------|
| Sample ratio mismatch (SRM) detected | Randomization bug, bot traffic, or redirects affecting one variant | Stop experiment immediately. Investigate: check for bot filtering differences, redirect loops, or caching issues. Fix root cause and restart with new experiment ID. |
| Experiment never reaches significance | Underpowered test — MDE too small for available traffic | Document as inconclusive. For next experiment, use larger MDE or increase traffic allocation. Consider CUPED variance reduction (20-50% sample size reduction). [src3] |
| Guardrail metric violated during experiment | Treatment has unintended negative side effect | If critical guardrail (revenue, error rate) — stop experiment. If non-critical — continue to full sample size, then decide. Document the guardrail violation. |
| Multiple teams running conflicting experiments | No experiment coordination or mutual exclusion | Implement experiment layers/namespaces in platform. Create shared experiment calendar. Assign traffic quotas per team. |
| Results contradict strong prior beliefs | Novelty effect, Hawthorne effect, or genuine surprise | Before acting, check for novelty effects by segmenting by exposure time. Consider holdback test (ship to 95%, hold 5% as control for 2 weeks). [src1] |
| Peeking at results caused premature stopping | Team member checked dashboard and made decision before sample size reached | Educate team on peeking problem. Switch to sequential testing method (always-valid p-values) or restrict dashboard access until sample size met. |

## Cost Breakdown

<!-- Real cost analysis so agents can warn users before execution. -->

| Component | Free Tier | Paid Tier | At Scale |
|-----------|-----------|-----------|----------|
| Experimentation platform | PostHog: 1M events/mo, Statsig: 10M events/mo | $150-500/mo (VWO, Optimizely) | $1,000-5,000/mo (enterprise) |
| Sample size calculator | Evan Miller: unlimited | Built-in (Statsig, Eppo) | Automated power analysis |
| Experiment tracker | Google Sheets: $0 | Notion/Confluence: $10-20/mo | Platform-native (Optimizely Collaboration) |
| Statistical analysis | Manual spreadsheet | Platform auto-analysis | Warehouse-native (Eppo, Statsig) |
| **Total per quarter** | **$0** | **$500-1,600** | **$3,000-15,000+** |

## Anti-Patterns

### Wrong: Peeking at results and stopping early
Running an experiment, checking the dashboard daily, and shipping the first time the p-value dips below 0.05. With daily peeking over a 30-day experiment, the actual false positive rate inflates from 5% to approximately 25-30%. One in four "winning" experiments is actually noise. [src1]

### Correct: Pre-commit to sample size or use sequential testing
Calculate sample size before launch and only analyze at completion. Alternatively, use a sequential testing framework (available in Statsig and Eppo) that maintains valid confidence intervals at every point, allowing early stopping with correct Type I error rates. [src3]

### Wrong: Testing too many things at once without correction
Running an experiment with 10 success metrics, then celebrating when any one of them shows significance. With 10 independent tests at alpha = 0.05, the probability of at least one false positive is 1 - (0.95^10) = 40%.

### Correct: Declare one primary metric and apply multiple testing correction
Choose a single Overall Evaluation Criterion (OEC) for the go/no-go decision. If monitoring secondary metrics, apply Bonferroni correction (divide alpha by number of metrics) or use false discovery rate control. [src1]

### Wrong: Running experiments on tiny audiences
Launching an A/B test on a feature used by 200 visitors/week with a 2% conversion rate. Even detecting a 50% relative lift (2% to 3%) requires approximately 4,700 per variation — 47 weeks of data.

### Correct: Match experiment ambition to available traffic
Use the sample size calculator BEFORE committing to an experiment. If the required runtime exceeds 8 weeks, either target a higher-traffic page, test a bolder change (larger MDE), or use qualitative methods instead. [src2]

## When This Matters

Use when a startup or product team needs a rigorous, repeatable system for running experiments — not ad hoc "let's try this and see what happens." This framework is essential once a product has sufficient traffic (100+ weekly conversions on the target metric) and the team is making decisions about product changes that could be validated with data. The framework prevents the two most expensive experiment failures: shipping losers that look like winners (false positives from peeking) and killing winners that looked inconclusive (underpowered tests).

## Related Units

- [Metric Definition Framework](/business/startup-metrics/metric-definition-framework) — defines the metrics experiments will measure
- [North Star Metric Selection](/business/startup-metrics/north-star-metric-selection) — determines the primary OEC for experiments
- [Growth Model Calibration](/business/startup-metrics/growth-model-calibration) — consumes experiment results to update growth projections
- [Qualitative Testing Guide](/business/startup-metrics/qualitative-testing-guide) — alternative for low-traffic products where A/B testing is infeasible