---
# === IDENTITY ===
id: software/landing-pages/ab-testing-infrastructure/2026
canonical_question: "How do I set up A/B testing — Posthog feature flags, VWO, custom JS split, minimum traffic for significance?"
aliases:
  - "PostHog A/B testing setup for landing pages"
  - "VWO landing page split test configuration"
  - "Minimum sample size for A/B test statistical significance"
  - "Custom JavaScript A/B split testing without third-party tools"
entity_type: execution_recipe
domain: software > landing-pages > A/B testing infrastructure
region: global
jurisdiction: global
temporal_scope: 2025-2026

# === VERIFICATION ===
last_verified: 2026-03-12
confidence: 0.88
version: 1.0
first_published: 2026-03-12

# === TEMPORAL VALIDITY ===
temporal_validity:
  status: evolving
  last_breaking_change: "PostHog experiments API v2 released mid-2025 with multivariate support; VWO restructured pricing tiers in late 2025"
  next_review: 2026-09-08
  change_sensitivity: high

# === CONSTRAINTS ===
constraints:
  - "Minimum sample size for 95% statistical significance with 5% MDE: ~3,800 visitors per variant at 5% baseline conversion rate"
  - "Running tests below minimum sample size produces unreliable results — do not declare winners before reaching calculated sample size"
  - "PostHog experiments free tier: 1M events/month — sufficient for most landing page tests"
  - "VWO Starter plan starts at ~$49/mo for up to 50K monthly tracked users"
  - "Custom JS splits must handle cookie persistence, bot filtering, and consistent variant assignment across sessions"
  - "Do not run multiple overlapping A/B tests on the same page element — interaction effects invalidate results"

# === SKIP CONDITIONS ===
skip_this_unit_if:
  - condition: "User needs analytics implementation, not A/B testing"
    use_instead: "software/landing-pages/analytics-implementation/2026"
  - condition: "User needs multivariate testing at enterprise scale"
    use_instead: "Search knowledgelib.io for landing page builder selection — no dedicated unit yet"
  - condition: "Landing page gets fewer than 1,000 visitors/month"
    use_instead: "software/landing-pages/form-lead-capture-setup/2026"

# === AGENT HINTS ===
inputs_needed:
  - key: tool_preference
    question: "Which A/B testing tool should be used?"
    type: choice
    options: ["PostHog", "VWO", "Google Optimize replacement", "Custom JavaScript", "no preference — auto-select"]
  - key: technical_skill
    question: "What is the user's technical skill level?"
    type: choice
    options: ["non-technical (no-code only)", "semi-technical (can edit code)", "developer (can write code)"]
  - key: budget_for_tools
    question: "What's the tool/API budget?"
    type: choice
    options: ["free tier only", "up to $50/month", "up to $200/month", "no limit"]
  - key: monthly_traffic
    question: "How many monthly visitors does the landing page receive?"
    type: choice
    options: ["under 1,000", "1,000-5,000", "5,000-20,000", "20,000-100,000", "100,000+"]

# === EXECUTION METADATA ===
execution:
  required_inputs:
    - name: "Deployed landing page with analytics"
      source: "software/landing-pages/analytics-implementation/2026"
      format: "live URL with GA4 or PostHog tracking active"
    - name: "Test hypothesis and variants"
      source: "user or optimization agent"
      format: "documented hypothesis with control and variant descriptions"
  outputs:
    - name: "Running A/B test"
      format: "live experiment"
      description: "Active split test distributing traffic between control and variant(s) with conversion tracking"
    - name: "Sample size calculation"
      format: "text document"
      description: "Required visitors per variant, estimated test duration, and significance parameters"
    - name: "Test results dashboard"
      format: "analytics dashboard URL"
      description: "Real-time experiment results showing conversion rates, confidence intervals, and significance status"
  tools_required:
    - name: "PostHog"
      purpose: "Feature flags, experiment management, and analytics"
      tier: "free"
      cost: "$0 (1M events/month free)"
      alternatives: ["VWO", "Statsig", "GrowthBook", "custom JavaScript"]
    - name: "VWO"
      purpose: "Visual A/B testing with drag-and-drop editor"
      tier: "paid"
      cost: "$49/mo Starter (50K users)"
      alternatives: ["PostHog", "Optimizely ($50K+/yr)", "AB Tasty"]
    - name: "Sample size calculator"
      purpose: "Determine minimum visitors needed per variant"
      tier: "free"
      cost: "$0"
      alternatives: ["Evan Miller calculator", "Optimizely calculator", "VWO calculator", "Statsig calculator"]
  credentials_needed:
    - service: "PostHog"
      type: "username+password or Google OAuth"
      where_to_get: "https://posthog.com/signup"
      free_tier_limits: "1M events/month, 1M feature flag requests/month"
    - service: "VWO"
      type: "username+password"
      where_to_get: "https://vwo.com/free-trial/"
      free_tier_limits: "30-day free trial, then $49/mo Starter"
  estimated_duration: "30-60 minutes for test setup + 2-8 weeks for results"
  estimated_cost: "$0 (PostHog free) to $49/mo (VWO Starter)"

# === DISTRIBUTION ===
canonical_source: "https://knowledgelib.io/software/landing-pages/ab-testing-infrastructure/2026"
suggested_citation: "Source: knowledgelib.io — AI Knowledge Library (verified 2026-03-12)"

# === RELATED UNITS ===
related_kos:
  depends_on:
    - id: "software/landing-pages/analytics-implementation/2026"
      label: "Analytics must be active to measure test results"
    - id: "software/landing-pages/form-lead-capture-setup/2026"
      label: "Form conversion events needed as test goals"
  feeds_into:
    - id: "software/landing-pages/performance-optimization-core-web-vitals/2026"
      label: "Test performance variants for CWV impact"
  alternative_to: []

# === SOURCES ===
sources:
  - id: src1
    title: "How to set up A/B/n testing — PostHog"
    author: PostHog
    url: https://posthog.com/tutorials/abn-testing
    type: official_docs
    published: 2025-01-01
    reliability: authoritative
  - id: src2
    title: "Feature flags — PostHog Docs"
    author: PostHog
    url: https://posthog.com/docs/feature-flags
    type: official_docs
    published: 2025-01-01
    reliability: authoritative
  - id: src3
    title: "How to Calculate A/B Test Sample Size — VWO"
    author: VWO
    url: https://vwo.com/blog/how-to-calculate-ab-test-sample-size/
    type: technical_blog
    published: 2025-01-01
    reliability: high
  - id: src4
    title: "A/B Test Sample Size Calculator — Optimizely"
    author: Optimizely
    url: https://www.optimizely.com/sample-size-calculator/
    type: official_docs
    published: 2025-01-01
    reliability: authoritative
  - id: src5
    title: "A/B Test Sample Size Calculator — Evan Miller"
    author: Evan Miller
    url: https://www.evanmiller.org/ab-testing/sample-size.html
    type: academic_tool
    published: 2025-01-01
    reliability: authoritative
  - id: src6
    title: "VWO Pricing 2026 — Free to $972/User Plans"
    author: CostBench
    url: https://costbench.com/software/ab-testing/vwo/
    type: industry_report
    published: 2026-01-01
    reliability: high
  - id: src7
    title: "7 Top Landing Page Optimization Tools 2026 — VWO"
    author: VWO
    url: https://vwo.com/blog/landing-page-optimization-tools/
    type: technical_blog
    published: 2026-01-01
    reliability: high
---

# A/B Testing Infrastructure

## Purpose

This recipe sets up a working A/B testing system on a landing page — from sample size calculation through test creation, variant deployment, and statistical analysis. The output is a running experiment that splits traffic between control and variant(s), tracks conversions per variant, and produces statistically valid results showing which version performs better. Covers three tool paths: PostHog (developer-friendly, free), VWO (visual editor, paid), and custom JavaScript (zero dependencies). [src1]

## Prerequisites

- [ ] **Deployed landing page with analytics** — [Analytics Implementation](/software/landing-pages/analytics-implementation/2026) (GA4 or PostHog must be tracking conversions)
- [ ] **Test hypothesis** — documented hypothesis with specific change and expected outcome (e.g., "Changing CTA from 'Get Started' to 'Start Free Trial' will increase form submissions by 10%")
- [ ] **Variant design** — mockup or description of what the variant looks like vs control
- [ ] **Monthly traffic estimate** — needed to calculate test duration
- [ ] **PostHog account** — [posthog.com/signup](https://posthog.com/signup) (free) or **VWO account** — [vwo.com/free-trial](https://vwo.com/free-trial/) (30-day trial)

## Constraints

- At 5% baseline conversion rate and 5% minimum detectable effect (MDE), you need ~3,800 visitors per variant for 95% significance and 80% power. With 1,000 monthly visitors, this test takes 8 months — likely too long to be useful. [src3] [src5]
- Never declare a winner before reaching calculated sample size. Early peeking inflates false positive rates from 5% to 20-30%. [src3]
- PostHog experiments free tier: 1M events/month — handles most landing page experiments without cost. [src2]
- VWO Starter: $49/mo for 50K monthly tracked users. [src6]
- Do not run multiple overlapping tests on the same page element. If testing headline AND CTA simultaneously, use a multivariate test, not two separate A/B tests.
- Client-side A/B testing (VWO, custom JS) can cause page flicker — the original content flashes before the variant loads. Mitigate with anti-flicker snippets or server-side rendering.

## Tool Selection Decision

```
Which path?
├── User is developer AND budget = free
│   └── PATH A: PostHog Experiments — free, feature flags + analytics in one
├── User is non-technical AND budget > $0
│   └── PATH B: VWO — $49/mo, visual drag-and-drop editor, no coding
├── User is developer AND wants zero dependencies
│   └── PATH C: Custom JavaScript — $0, lightweight split with GA4 events
└── User has high traffic AND enterprise needs
    └── PATH D: Statsig/Optimizely — $0-50K/yr, advanced stats engine
```

| Path | Tools | Cost | Setup Time | Best For |
|------|-------|------|------------|----------|
| A: PostHog | PostHog Cloud | $0 | 30 min | Developers, product teams |
| B: VWO | VWO Starter | $49/mo | 20 min | Non-technical, visual tests |
| C: Custom JS | Vanilla JS + GA4 | $0 | 45 min | Minimalists, privacy-focused |
| D: Enterprise | Statsig/Optimizely | $0-50K+/yr | 60 min | High traffic, advanced stats |

## Execution Flow

### Step 1: Calculate Required Sample Size

**Duration**: 5 minutes
**Tool**: Online calculator

Before creating the test, calculate how many visitors you need and how long the test will take.

```
Sample Size Calculator Inputs:
─────────────────────────────────────────
Baseline conversion rate:  Your current conversion rate (e.g., 5%)
Minimum detectable effect: Smallest improvement worth detecting (e.g., 20% relative = 5% → 6%)
Statistical significance:  95% (standard)
Statistical power:         80% (standard)

Quick Reference (95% significance, 80% power):
─────────────────────────────────────────
Baseline 2%, MDE 50% relative (2% → 3%):    ~3,600 per variant
Baseline 5%, MDE 20% relative (5% → 6%):    ~3,800 per variant
Baseline 5%, MDE 50% relative (5% → 7.5%):  ~700 per variant
Baseline 10%, MDE 20% relative (10% → 12%): ~1,900 per variant
Baseline 10%, MDE 50% relative (10% → 15%): ~350 per variant

Test Duration = (Visitors per variant × Number of variants) / Monthly traffic
Example: 3,800 × 2 variants / 10,000 monthly visitors = 0.76 months ≈ 23 days
```

Use [Evan Miller's calculator](https://www.evanmiller.org/ab-testing/sample-size.html) or [Optimizely's calculator](https://www.optimizely.com/sample-size-calculator/) for precise numbers. [src4] [src5]

**Verify**: Test duration is less than 8 weeks. If longer, increase MDE (accept detecting only larger improvements) or increase traffic first.
**If failed**: If test would take > 3 months, consider qualitative testing (user interviews, heatmap analysis) instead of quantitative A/B testing.

### Step 2: Create the Experiment

**Duration**: 10-20 minutes
**Tool**: PostHog, VWO, or code editor

**Path A (PostHog):**

1. In PostHog, go to Experiments → New Experiment
2. Set the feature flag key (e.g., `landing-page-cta-test`)
3. Add variants: Control (existing) and Test (new CTA text)
4. Set the goal metric: the conversion event you want to improve (e.g., `generate_lead`)
5. Set minimum acceptable improvement (your MDE from Step 1)
6. Set participant allocation (usually 50/50) [src1]

Add the PostHog experiment code to your landing page:

```javascript
// Check which variant the user is in
posthog.onFeatureFlags(function() {
  const variant = posthog.getFeatureFlag('landing-page-cta-test');

  if (variant === 'test') {
    // Apply variant changes
    document.querySelector('.cta-button').textContent = 'Start Free Trial';
    document.querySelector('.hero-subhead').textContent = 'No credit card required. Set up in 2 minutes.';
  }
  // Control: do nothing (original page serves as control)
});
```

**Path B (VWO):**

1. In VWO, create a new A/B Test
2. Enter your landing page URL
3. Use the Visual Editor to modify elements (change text, colors, images, layout)
4. VWO automatically serves the original as control
5. Set the primary goal (URL visit, click, custom event)
6. Set traffic allocation (50/50 recommended)
7. Start the test — VWO injects changes via its JavaScript snippet [src7]

**Path C (Custom JavaScript):**

```javascript
// Lightweight A/B split — no dependencies
(function() {
  const TEST_NAME = 'cta-test-v1';
  const VARIANTS = ['control', 'variant-a'];

  // Get or assign variant (persisted in localStorage)
  let variant = localStorage.getItem('ab_' + TEST_NAME);
  if (!variant) {
    variant = VARIANTS[Math.floor(Math.random() * VARIANTS.length)];
    localStorage.setItem('ab_' + TEST_NAME, variant);
  }

  // Apply variant changes
  if (variant === 'variant-a') {
    document.addEventListener('DOMContentLoaded', function() {
      const cta = document.querySelector('.cta-button');
      if (cta) cta.textContent = 'Start Free Trial';
    });
  }

  // Send variant info to GA4
  if (window.gtag) {
    gtag('event', 'experiment_impression', {
      experiment_id: TEST_NAME,
      variant_id: variant
    });
  }
})();
```

**Verify**: Open the landing page in two different incognito windows. One should show the control, one should show the variant (due to random assignment). Check PostHog/VWO/GA4 for experiment impression events.
**If failed**: If both windows show the same variant, clear localStorage/cookies between tests. For PostHog, check that the feature flag is active and the experiment is running.

### Step 3: Implement Anti-Flicker Protection

**Duration**: 5 minutes
**Tool**: Code editor

Client-side A/B testing causes page flicker — the original content shows for a fraction of a second before the variant loads. This degrades user experience and can bias results.

```html
<!-- Anti-flicker snippet — add BEFORE the testing tool script in <head> -->
<style>
  .ab-testing-hide { opacity: 0 !important; }
</style>
<script>
  // Hide elements being tested until variant is applied
  document.documentElement.classList.add('ab-testing-hide');
  // Fallback: show page after 2 seconds even if test script fails
  setTimeout(function() {
    document.documentElement.classList.remove('ab-testing-hide');
  }, 2000);
</script>

<!-- Then in your test code, after applying variant: -->
<script>
  // Remove the hide class after variant is applied
  document.documentElement.classList.remove('ab-testing-hide');
</script>
```

For PostHog, use server-side evaluation (bootstrapping) to avoid flicker entirely. For VWO, enable the built-in anti-flicker setting in campaign settings.

**Verify**: Load the page with throttled network (Chrome DevTools → Network → Slow 3G). The page should not flash the original content before showing the variant.
**If failed**: If flicker persists, move the anti-flicker script higher in `<head>` or increase the timeout to 3 seconds.

### Step 4: Set Up Conversion Goal Tracking

**Duration**: 5-10 minutes
**Tool**: PostHog, VWO, or GA4

Ensure the conversion event is properly tracked for each variant.

**PostHog**: The goal metric was set in Step 2. PostHog automatically tracks conversions per variant. Add secondary metrics if needed (e.g., page scroll depth, time on page). [src1]

**VWO**: Set the primary goal in the test settings. Options: page visit (thank you page), element click (CTA button), or custom event.

**Custom JS + GA4**: Track conversions with experiment context:

```javascript
// On form submission or CTA click
document.querySelector('.cta-button').addEventListener('click', function() {
  const variant = localStorage.getItem('ab_cta-test-v1') || 'control';
  gtag('event', 'generate_lead', {
    experiment_id: 'cta-test-v1',
    variant_id: variant
  });
});
```

**Verify**: Submit a test conversion in each variant. Check that the conversion event includes the variant identifier in PostHog/VWO/GA4.
**If failed**: If conversions are not attributed to variants, check that the experiment impression fires before the conversion event.

### Step 5: Monitor and Wait for Significance

**Duration**: 2-8 weeks (passive monitoring)
**Tool**: PostHog, VWO, or significance calculator

**Check results weekly** (but do not stop the test early):

```
Weekly Monitoring Checklist:
─────────────────────────────────────────
1. Traffic split is approximately 50/50 (±5%)
2. No technical errors in either variant (check error logs)
3. Sample size accumulating on track for estimated completion date
4. Conversion rate for both variants is within expected range
5. No external events biasing results (product launch, press mention)
```

**PostHog**: The experiment dashboard shows a progress bar toward significance and a Bayesian probability of each variant being the winner. [src1]

**VWO**: The campaign report shows statistical significance percentage and whether a winner can be declared. [src7]

**Custom JS**: Use [VWO's significance calculator](https://vwo.com/tools/ab-test-significance-calculator/) — enter visitors and conversions for each variant.

**Verify**: At the calculated sample size, one variant has > 95% significance. If neither reaches 95%, the difference is not statistically significant — keep the control.
**If failed**: If after reaching full sample size neither variant wins, the MDE was too small to detect. Either test a bigger change or increase traffic volume for the next test.

### Step 6: Implement the Winner and Clean Up

**Duration**: 10 minutes
**Tool**: Code editor + testing tool

Once a winner is declared:

1. Implement the winning variant as the permanent default on the landing page
2. Remove the A/B testing code and anti-flicker snippet
3. Archive the experiment in PostHog/VWO for future reference
4. Document the result: hypothesis, variants, sample size, conversion rates, significance level, and decision

```
Test Result Documentation Template:
─────────────────────────────────────────
Test Name:        cta-test-v1
Hypothesis:       "Changing CTA to 'Start Free Trial' increases conversions"
Duration:         23 days (March 12 - April 4, 2026)
Sample Size:      7,847 total (3,921 control, 3,926 variant)
Control CVR:      4.8%
Variant CVR:      6.1%
Relative Lift:    +27.1%
Significance:     97.3% (> 95% threshold)
Decision:         IMPLEMENT VARIANT
```

**Verify**: The winning variant is now the default page. All A/B testing code is removed. Conversion rate remains stable for 1 week post-implementation.
**If failed**: If conversion rate drops after implementation, check that the permanent implementation matches the test variant exactly (same copy, same styling, same functionality).

## Output Schema

```json
{
  "output_type": "ab_test_configuration",
  "format": "running experiment + documentation",
  "columns": [
    {"name": "test_name", "type": "string", "description": "Unique test identifier", "required": true},
    {"name": "tool", "type": "string", "description": "Testing platform (PostHog/VWO/Custom JS)", "required": true},
    {"name": "variants", "type": "array", "description": "List of variants with descriptions", "required": true},
    {"name": "goal_metric", "type": "string", "description": "Primary conversion event being measured", "required": true},
    {"name": "sample_size_required", "type": "number", "description": "Minimum visitors per variant", "required": true},
    {"name": "estimated_duration", "type": "string", "description": "Weeks to reach significance", "required": true},
    {"name": "status", "type": "string", "description": "running/completed/stopped", "required": true}
  ],
  "expected_row_count": "1",
  "sort_order": "N/A",
  "deduplication_key": "test_name"
}
```

## Quality Benchmarks

| Quality Metric | Minimum Acceptable | Good | Excellent |
|---------------|-------------------|------|-----------|
| Traffic split accuracy | Within ±10% of target | Within ±5% | Within ±2% |
| Sample size reached | > 80% of calculated minimum | 100% of minimum | 120% of minimum |
| Test duration | < 12 weeks | < 6 weeks | < 3 weeks |
| Flicker-free experience | Anti-flicker < 2s delay | Anti-flicker < 500ms | Server-side (zero flicker) |
| Result documentation | Winner declared | Full stats documented | Learnings shared with team |

**If below minimum**: If traffic split is off by > 10%, check feature flag configuration. If sample size is unreachable, increase MDE or increase traffic. [src3]

## Error Handling

| Error | Likely Cause | Recovery Action |
|-------|-------------|----------------|
| Both variants show same content | Feature flag not evaluating or script error | Check PostHog/VWO dashboard for flag status; verify JavaScript loads without errors |
| Conversion rates are identical at 0% | Conversion events not firing or not attributed to variants | Verify conversion event fires with variant identifier; check event name matches goal metric |
| Page flicker visible | Anti-flicker snippet missing or loading too late | Move anti-flicker CSS to first element in `<head>`; use server-side rendering |
| Traffic not splitting evenly | Feature flag distribution issue or caching | Check CDN caching is not serving same variant to all users; verify flag percentage in PostHog |
| Test running but PostHog shows no data | PostHog script not loading or blocked | Check browser console for PostHog errors; test in incognito; verify API key is correct |
| Significance fluctuating up and down | Normal statistical behavior before reaching sample size | Do not stop test early; wait until full sample size is reached per Step 1 calculation |

## Cost Breakdown

| Component | Free Tier | Paid Tier | At Scale |
|-----------|-----------|-----------|----------|
| PostHog Experiments | $0 (1M events/mo) | Pay-per-use beyond free | $0.00005/event beyond 1M |
| VWO | $0 (30-day trial) | $49/mo Starter (50K users) | $972/mo Enterprise |
| Statsig | $0 (50M events/mo) | $150/mo Pro | Custom Enterprise |
| Custom JS + GA4 | $0 | $0 | $0 |
| Significance calculators | $0 | $0 | $0 |
| **Total** | **$0** | **$0-49/mo** | **$49-972/mo** |

## Anti-Patterns

### Wrong: Stopping the test when one variant "looks like it's winning"
Peeking at results and stopping early inflates false positive rates from 5% to over 25%. A variant that looks 15% better at day 3 may end up identical to control at day 23. [src3]

### Correct: Pre-commit to the calculated sample size
Before starting the test, calculate required sample size (Step 1) and commit to running until that number is reached. Only evaluate significance at the pre-determined stopping point.

### Wrong: Testing tiny changes on low-traffic pages
Testing button color changes on a page with 500 monthly visitors requires 15+ months to detect a 20% relative improvement. The test will never reach significance. [src5]

### Correct: Match test ambition to traffic volume
Low-traffic pages (< 5K/month): test big changes only (entirely different headlines, layouts, offers). High-traffic pages (> 50K/month): can test subtle changes (button copy, social proof placement).

## When This Matters

Use this recipe when the agent needs to set up a quantitative A/B test on a landing page to compare variant performance. Requires an existing landing page with analytics tracking and conversion events already configured. This recipe handles experiment design, tool setup, and statistical analysis — it does not generate variant ideas or design the variants themselves.

## Related Units

- [Analytics Implementation](/software/landing-pages/analytics-implementation/2026) — analytics must be active before running tests
- [Form Lead Capture Setup](/software/landing-pages/form-lead-capture-setup/2026) — form conversion events serve as test goals
- [Performance Optimization: Core Web Vitals](/software/landing-pages/performance-optimization-core-web-vitals/2026) — test performance variants
- [Startup Analytics Implementation Guide](/business/startup-metrics/startup-analytics-implementation-guide/2026) — broader experimentation strategy