Global DNS and Load Balancing Architecture

TL;DR

• Bottom line: Global DNS load balancing routes users to the nearest healthy backend using a combination of GeoDNS, Anycast, and health-checked weighted records — choose your strategy based on latency requirements, compliance constraints, and infrastructure topology.
• Key tool/command: aws route53 create-health-check + latency/geolocation routing policies (or Cloudflare Load Balancing with steering policies)
• Watch out for: Setting DNS TTL too low (causes query floods) or too high (causes slow failover) — 60-300s is the sweet spot for most GSLB deployments.
• Works with: AWS Route 53, Cloudflare, Google Cloud DNS, Azure Traffic Manager, HAProxy ALOHA, NGINX Plus, PowerDNS, NS1.

Constraints

• DNS TTL floor: Never set TTL below 30s — resolver caching varies wildly and ultra-low TTLs increase query volume 10-50x without proportional failover benefit
• Anycast requires owning/leasing a /24 IPv4 or /48 IPv6 block and coordinating BGP with every datacenter provider
• Health checks must run from multiple geographic vantage points (minimum 3 regions) to avoid false positives from network partitions
• DNS-only load balancing cannot enforce session affinity — combine with L7 LB or cookie persistence when stateful sessions are required
• GeoDNS accuracy degrades with VPN/proxy users and resolvers that do not support EDNS Client Subnet (ECS)

Quick Reference

Decision Tree

START
├── Need data sovereignty / compliance routing?
│   ├── YES → Use GeoDNS with jurisdiction-locked regions
│   │         (Route 53 Geolocation or Cloudflare Geo Steering)
│   └── NO ↓
├── Need sub-100ms failover with zero DNS propagation delay?
│   ├── YES → Use Anycast IP with L7 health checks
│   │         (GCP Global LB, Cloudflare proxy, custom BGP)
│   └── NO ↓
├── Need lowest-latency routing across 3+ regions?
│   ├── YES → Use latency-based routing with health checks
│   │         (Route 53 Latency + failover, Cloudflare Dynamic Steering)
│   └── NO ↓
├── Need canary/blue-green deployments?
│   ├── YES → Use weighted routing (e.g., 90/10 split)
│   │         (Route 53 Weighted, Cloudflare Weighted Steering)
│   └── NO ↓
├── <10K req/s with 2-3 regions?
│   ├── YES → Simple DNS round-robin with health checks
│   │         (Any managed DNS provider)
│   └── NO ↓
└── DEFAULT → Combine Anycast (edge) + GeoDNS (regional) + L7 LB (local)
              This is the standard architecture for >100K req/s global services

Step-by-Step Guide

1. Define regions and deploy backends

Identify 2-6 geographic regions based on your user distribution. Deploy identical application stacks in each region with independent databases or read replicas. [src1]

# Example: List your target regions and endpoints
# Region: us-east-1  → alb-us-east.example.com
# Region: eu-west-1  → alb-eu-west.example.com
# Region: ap-south-1 → alb-ap-south.example.com

Verify: curl -s -o /dev/null -w "%{http_code}" https://alb-us-east.example.com/health → expected: 200

2. Configure health checks from multiple vantage points

Set up health checks that probe each backend from at least 3 geographic locations. Configure failure thresholds (typically 3 consecutive failures) and check intervals (10-30s). [src2]

# AWS Route 53: Create health check
aws route53 create-health-check --caller-reference "us-east-$(date +%s)" \
  --health-check-config '{
    "FullyQualifiedDomainName": "alb-us-east.example.com",
    "Port": 443,
    "Type": "HTTPS",
    "ResourcePath": "/health",
    "RequestInterval": 10,
    "FailureThreshold": 3,
    "Regions": ["us-east-1", "eu-west-1", "ap-southeast-1"]
  }'

Verify: aws route53 get-health-check-status --health-check-id <ID> → all reporters show Success

3. Set up DNS routing policy

Choose and configure your primary routing strategy (latency, geolocation, or weighted). Associate health checks with each record set. [src2]

# AWS Route 53: Latency-based routing with health check
aws route53 change-resource-record-sets --hosted-zone-id Z1234567890 \
  --change-batch '{
    "Changes": [{
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "api.example.com",
        "Type": "A",
        "SetIdentifier": "us-east-1",
        "Region": "us-east-1",
        "TTL": 60,
        "ResourceRecords": [{"Value": "203.0.113.10"}],
        "HealthCheckId": "hc-us-east-id"
      }
    }]
  }'

Verify: dig api.example.com +short from different regions returns region-appropriate IPs

4. Configure local load balancers per region

In each region, deploy L4/L7 load balancers (NGINX, HAProxy, or cloud ALBs) to distribute traffic across local backend instances. [src4]

frontend http_front
    bind *:443 ssl crt /etc/ssl/certs/example.pem
    default_backend app_servers

backend app_servers
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    server app1 10.0.1.10:8080 check inter 5s fall 3 rise 2
    server app2 10.0.1.11:8080 check inter 5s fall 3 rise 2
    server app3 10.0.1.12:8080 check inter 5s fall 3 rise 2

Verify: echo "show stat" | socat /var/run/haproxy.sock stdio | grep app_servers → all servers show UP

5. Implement failover hierarchy

Configure primary, secondary, and tertiary failover targets so that regional failures cascade to the next-best region automatically. [src1]

# Cloudflare: Configure failover pool priority via API
curl -X PATCH "https://api.cloudflare.com/client/v4/zones/{zone_id}/load_balancers/{lb_id}" \
  -H "Authorization: Bearer $CF_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{
    "default_pools": ["pool-us-east", "pool-eu-west", "pool-ap-south"],
    "fallback_pool": "pool-us-west",
    "steering_policy": "dynamic_latency",
    "session_affinity": "cookie"
  }'

Verify: Disable primary pool health endpoint, then dig example.com → should return secondary pool IPs within TTL window

6. Set up monitoring and alerting

Monitor DNS resolution times, failover events, health check flaps, and regional traffic distribution. Alert on anomalies. [src6]

# Prometheus alert rule: DNS failover detected
alert: DNSFailoverTriggered
expr: probe_dns_lookup_time_seconds > 1
for: 2m
labels:
  severity: warning
annotations:
  summary: "DNS failover detected for {{ $labels.instance }}"

Verify: Trigger a test failover and confirm alerts fire within expected thresholds

Code Examples

Terraform: Route 53 Latency-Based Routing with Health Checks

# Input:  Multi-region ALB endpoints
# Output: Latency-routed DNS with automatic failover

resource "aws_route53_health_check" "us_east" {
  fqdn              = "alb-us-east.example.com"
  port               = 443
  type               = "HTTPS"
  resource_path      = "/health"
  failure_threshold  = 3
  request_interval   = 10
  regions            = ["us-east-1", "eu-west-1", "ap-southeast-1"]

  tags = { Name = "us-east-health-check" }
}

resource "aws_route53_record" "api_us_east" {
  zone_id         = aws_route53_zone.main.zone_id
  name            = "api.example.com"
  type            = "A"
  set_identifier  = "us-east-1"
  ttl             = 60

  latency_routing_policy {
    region = "us-east-1"
  }

  records          = ["203.0.113.10"]
  health_check_id  = aws_route53_health_check.us_east.id
}

HAProxy: GSLB with DNS-Based Backend Resolution

# Input:  DNS-discovered backend servers across regions
# Output: Load-balanced traffic with health-aware routing

resolvers cloudns
    nameserver dns1 8.8.8.8:53
    nameserver dns2 1.1.1.1:53
    resolve_retries 3
    timeout resolve 1s
    timeout retry   1s
    hold valid      60s
    accepted_payload_size 8192

frontend gslb_front
    bind *:443 ssl crt /etc/ssl/certs/example.pem
    default_backend regional_backends

backend regional_backends
    balance leastconn
    option httpchk GET /health
    http-check expect status 200
    server-template us-east- 3 us-east.example.com:443 \
        check inter 10s fall 3 rise 2 ssl verify required \
        resolvers cloudns resolve-prefer ipv4
    server-template eu-west- 3 eu-west.example.com:443 \
        check inter 10s fall 3 rise 2 ssl verify required \
        resolvers cloudns resolve-prefer ipv4

Python: Programmatic DNS Health Check Monitor

# Input:  List of endpoints and regions
# Output: Health status map with latency metrics

import dns.resolver  # dnspython==2.6.1
import requests      # requests==2.31.0
import time
from concurrent.futures import ThreadPoolExecutor

ENDPOINTS = {
    "us-east": "alb-us-east.example.com",
    "eu-west": "alb-eu-west.example.com",
    "ap-south": "alb-ap-south.example.com",
}

def check_endpoint(region, host):
    try:
        start = time.monotonic()
        answers = dns.resolver.resolve(host, "A")
        dns_ms = (time.monotonic() - start) * 1000
        start = time.monotonic()
        resp = requests.get(f"https://{host}/health", timeout=5)
        http_ms = (time.monotonic() - start) * 1000
        return {"region": region, "status": "healthy" if resp.status_code == 200 else "degraded",
                "dns_ms": round(dns_ms, 2), "http_ms": round(http_ms, 2)}
    except Exception as e:
        return {"region": region, "status": "unhealthy", "error": str(e)}

with ThreadPoolExecutor(max_workers=len(ENDPOINTS)) as pool:
    results = list(pool.map(lambda kv: check_endpoint(*kv), ENDPOINTS.items()))
for r in results:
    print(f"[{r['region']}] {r['status']} dns={r.get('dns_ms','N/A')}ms http={r.get('http_ms','N/A')}ms")

Anti-Patterns

Wrong: Single-region health check for global DNS

# BAD — health check from one location only
# A network partition between probe and backend triggers false failover
health_check:
  type: HTTPS
  endpoint: alb-us-east.example.com
  probe_region: us-east-1  # Single point of failure!
  failure_threshold: 1     # Too aggressive!

Correct: Multi-region health checks with sensible thresholds

# GOOD — probes from 3+ regions, requires consensus
health_check:
  type: HTTPS
  endpoint: alb-us-east.example.com
  probe_regions: [us-east-1, eu-west-1, ap-southeast-1]
  failure_threshold: 3
  request_interval: 10

Wrong: DNS TTL of 0 or 5 seconds for "instant" failover

# BAD — TTL=5 causes massive query amplification
# Resolvers may ignore TTL < 30s anyway (RFC 8767)
api.example.com.  5  IN  A  203.0.113.10
# Result: 100x query volume, no real failover improvement

Correct: TTL 60-300s balanced with active health checks

# GOOD — TTL=60 provides fast-enough failover with manageable query volume
api.example.com.  60  IN  A  203.0.113.10
# Combined with Route 53 health checks: actual failover time = ~70-90s

Wrong: GeoDNS without fallback for unmapped regions

# BAD — users from unmapped regions get NXDOMAIN or random routing
geo_routing = {
    "US": "203.0.113.10",
    "EU": "198.51.100.10",
    # No default! Users from Africa, South America get nothing
}

Correct: GeoDNS with explicit default fallback

# GOOD — every region has a path, unmapped locations get best-effort routing
geo_routing = {
    "US": "203.0.113.10",
    "EU": "198.51.100.10",
    "AP": "192.0.2.10",
    "DEFAULT": "203.0.113.10",  # Fallback to lowest-latency PoP
}

Wrong: Anycast with long-lived TCP without session persistence

# BAD — BGP route changes mid-connection cause TCP resets
# Anycast works for DNS (UDP) and short HTTP, but long-lived
# WebSocket/gRPC streams break on route flaps
anycast_ip: 198.51.100.1 → backend-pool (no session tracking)

Correct: Anycast for initial connection + session affinity for stateful protocols

# GOOD — Anycast handles initial routing, L7 LB pins the session
Client → Anycast IP (198.51.100.1)
       → Nearest edge PoP (BGP)
       → L7 Load Balancer (session cookie / consistent hashing)
       → Specific backend instance (pinned for connection lifetime)

Common Pitfalls

• TTL caching ignored by resolvers: Some ISP resolvers cache DNS records beyond the stated TTL (RFC 8767 allows stale serving). Fix: design for worst-case TTL of 2x your configured value; use Anycast if faster failover needed. [src1]
• EDNS Client Subnet (ECS) not supported: When the resolver does not pass ECS, GeoDNS routes based on resolver location, not user location. Fix: use Anycast as primary strategy; GeoDNS as secondary. [src7]
• Health check flapping causing DNS churn: An overloaded backend oscillates between healthy/unhealthy, causing rapid DNS changes. Fix: set rise threshold (e.g., 3 consecutive successes) before re-adding; implement circuit breakers. [src4]
• Forgetting to pre-lower TTL before migration: Changing DNS records while TTL is 3600s means old records persist for up to 1 hour. Fix: lower TTL to 60s at least 48 hours before any DNS migration. [src2]
• Single DNS provider as sole authority: If Route 53 has an outage, entire global routing fails. Fix: use multi-provider DNS (Route 53 + Cloudflare, or NS1 + Google Cloud DNS). [src6]
• Not testing failover under real conditions: Failover works in staging but thundering herd effects appear in production. Fix: run regular chaos engineering drills during low-traffic windows. [src3]
• DNS propagation ignored in CI/CD: Deploy pipeline routes traffic to a region before DNS has propagated. Fix: deploy code first, verify health checks pass, then update DNS routing. [src1]
• Asymmetric capacity across regions: GeoDNS sends equal traffic to all regions but one has half the capacity. Fix: use weighted routing to match traffic to provisioned capacity; implement auto-scaling. [src2]

Diagnostic Commands

# Check DNS resolution from multiple resolvers
dig api.example.com +short @8.8.8.8
dig api.example.com +short @1.1.1.1

# Trace full DNS resolution path
dig api.example.com +trace

# Check DNS response with ECS (EDNS Client Subnet)
dig api.example.com +subnet=198.51.100.0/24 @8.8.8.8

# Verify TTL values
dig api.example.com +noall +answer

# Test health check endpoint
curl -s -o /dev/null -w "HTTP %{http_code} in %{time_total}s\n" https://api.example.com/health

# Check Route 53 health check status
aws route53 get-health-check-status --health-check-id <HC_ID>

# List Cloudflare load balancer pool health
curl -s "https://api.cloudflare.com/client/v4/zones/{zone}/load_balancers/pools/{pool}/health" \
  -H "Authorization: Bearer $CF_TOKEN" | jq '.result'

# Monitor DNS resolution latency over time
while true; do echo "$(date +%T) $(dig api.example.com +noall +stats | grep 'Query time')"; sleep 10; done

# Check BGP route for Anycast IP
whois -h whois.radb.net 198.51.100.0/24

Version History & Compatibility

When to Use / When Not to Use

Important Caveats

• DNS is fundamentally a best-effort routing layer — you cannot guarantee all clients will follow your TTL or routing hints (corporate proxies, legacy resolvers, and mobile networks may cache aggressively)
• Anycast provides near-instant failover for stateless protocols (DNS, HTTP) but can cause TCP connection resets during BGP convergence for long-lived connections
• Cloud provider GSLB services (Route 53, Cloudflare LB, GCP GLB) are easier to operate than self-managed Anycast/GeoDNS, but create vendor lock-in at the DNS layer
• Health checks consume backend resources — at scale (many probes x many regions x short intervals), health check traffic itself can be significant
• Route 53 latency-based routing measures latency to AWS regions, not to your specific backends — actual user experience may differ