--- # === IDENTITY === id: software/system-design/logging-monitoring/2026 canonical_question: "How do I design a logging and monitoring infrastructure?" aliases: - "How to build an observability stack for production systems" - "ELK vs Loki vs cloud-native monitoring comparison" - "What is the best logging and monitoring architecture" - "How to set up centralized logging with metrics and traces" entity_type: software_reference domain: software > system-design > logging_monitoring region: global jurisdiction: global temporal_scope: 2020-2026 # === VERIFICATION === last_verified: 2026-02-23 confidence: 0.93 version: 1.0 first_published: 2026-02-23 # === TEMPORAL VALIDITY === temporal_validity: status: stable last_breaking_change: "OpenTelemetry Logs GA (2024-04)" next_review: 2026-08-22 change_sensitivity: medium # === CONSTRAINTS === constraints: - "Never log secrets, tokens, passwords, PII, or credit card numbers — sanitize before emission" - "Retention policies must comply with local data regulations (GDPR 30-day default, SOC2 1-year minimum)" - "Always use structured (JSON) logging in production — unstructured text breaks automated parsing" - "Instrument with OpenTelemetry SDK when possible — vendor-neutral, prevents lock-in" - "Set per-service log-level controls — DEBUG in production for all services will overwhelm storage" # === SKIP CONDITIONS === skip_this_unit_if: - condition: "Looking for application performance monitoring (APM) tool comparison only" use_instead: "software/system-design/apm-comparison/2026" - condition: "Need to set up alerting rules and on-call rotations specifically" use_instead: "software/system-design/incident-management/2026" # === AGENT HINTS === inputs_needed: - key: "scale" question: "What is your expected log volume per day?" type: choice options: ["<10 GB/day", "10-100 GB/day", "100 GB-1 TB/day", ">1 TB/day"] - key: "environment" question: "What is your deployment environment?" type: choice options: ["Kubernetes", "VMs/bare-metal", "serverless", "hybrid"] - key: "budget_preference" question: "Do you prefer open-source self-hosted or managed SaaS?" type: choice options: ["open-source", "managed-saas", "hybrid"] # === DISTRIBUTION === canonical_source: "https://knowledgelib.io/software/system-design/logging-monitoring/2026" suggested_citation: "Source: knowledgelib.io — AI Knowledge Library (verified 2026-02-23)" # === RELATED UNITS === related_kos: depends_on: [] related_to: - id: "software/system-design/apm-comparison/2026" label: "APM Tool Comparison" - id: "software/system-design/distributed-tracing/2026" label: "Distributed Tracing Architecture" solves: - id: "software/system-design/incident-response/2026" label: "Incident Response and Debugging" alternative_to: [] often_confused_with: - id: "software/system-design/alerting-oncall/2026" label: "Alerting and On-Call Rotation Design" # === SOURCES (7 authoritative sources) === sources: - id: src1 title: "OpenTelemetry Documentation — Observability Primer" author: OpenTelemetry Project (CNCF) url: https://opentelemetry.io/docs/concepts/observability-primer/ type: official_docs published: 2025-12-01 reliability: authoritative - id: src2 title: "Prometheus Overview — Architecture and Components" author: Prometheus Authors url: https://prometheus.io/docs/introduction/overview/ type: official_docs published: 2025-10-01 reliability: authoritative - id: src3 title: "The Complete Guide to the ELK Stack" author: Logz.io url: https://logz.io/learn/complete-guide-elk-stack/ type: technical_blog published: 2025-06-15 reliability: high - id: src4 title: "Grafana Loki vs. ELK Stack for Logging: A Comprehensive Comparison" author: OpsVerse url: https://opsverse.io/2024/07/26/grafana-loki-vs-elk-stack-for-logging-a-comprehensive-comparison/ type: technical_blog published: 2024-07-26 reliability: moderate_high - id: src5 title: "Three Pillars of Observability: Metrics, Traces & Logs" author: Datadog url: https://www.datadoghq.com/three-pillars-of-observability/ type: technical_blog published: 2025-03-01 reliability: high - id: src6 title: "OpenTelemetry Collector Configuration" author: OpenTelemetry Project (CNCF) url: https://opentelemetry.io/docs/collector/configuration/ type: official_docs published: 2026-02-05 reliability: authoritative - id: src7 title: "Monitoring Anti-Patterns — Practical Monitoring (O'Reilly)" author: Mike Julian url: https://www.oreilly.com/library/view/practical-monitoring/9781491957349/ch01.html type: technical_blog published: 2024-01-01 reliability: high --- # Logging and Monitoring Infrastructure Design ## TL;DR - **Bottom line**: A production-grade observability stack requires three pillars — structured logs, time-series metrics, and distributed traces — unified through OpenTelemetry and stored in purpose-built backends (ELK/Loki for logs, Prometheus/Mimir for metrics, Jaeger/Tempo for traces). - **Key tool/command**: `otel-collector` (OpenTelemetry Collector) as the universal telemetry pipeline — receives, processes, and exports all three signal types. - **Watch out for**: Logging everything at DEBUG level in production — it will overwhelm storage, spike costs, and obscure real issues in noise. - **Works with**: Any language with OpenTelemetry SDK support (Go, Java, Python, Node.js, .NET, Rust, C++, Ruby, PHP, Swift, Erlang); Kubernetes and VM deployments; all major cloud providers. ## Constraints - Never log secrets, tokens, passwords, PII, or credit card numbers — sanitize all sensitive fields before emission - Retention policies must comply with local data regulations (GDPR defaults to 30-day minimum, SOC2 requires 1-year retention minimum) - Always use structured (JSON) logging in production — unstructured plaintext breaks automated parsing, indexing, and alerting - Instrument with OpenTelemetry SDK when possible — it is vendor-neutral and prevents backend lock-in - Set per-service log-level controls — global DEBUG across all services will cost 5-10x more in storage and processing ## Quick Reference | Component | Role | Technology Options | Scaling Strategy | |---|---|---|---| | Log Collection Agent | Ships logs from hosts/containers to aggregator | Fluent Bit, Filebeat, Vector, OTel Collector | DaemonSet per node (K8s) or sidecar | | Log Aggregation | Centralizes, parses, enriches log streams | Logstash, Fluentd, OTel Collector | Horizontal replicas behind buffer (Kafka) | | Log Storage & Search | Indexes and queries log data | Elasticsearch, Grafana Loki, ClickHouse | Elasticsearch: shard-per-index; Loki: label-based partitioning | | Metrics Collection | Scrapes/receives numeric time-series data | Prometheus, OTel Collector, Telegraf | Federation or Thanos/Mimir for multi-cluster | | Metrics Storage | Long-term time-series persistence | Prometheus TSDB, Thanos, Mimir, VictoriaMetrics | Remote-write to durable store; compaction + downsampling | | Trace Collection | Captures distributed request spans | OTel SDK + Collector, Jaeger Agent | Tail-based sampling at Collector tier | | Trace Storage | Stores and indexes span data | Jaeger, Grafana Tempo, Zipkin | Tempo: object storage (S3/GCS); Jaeger: Elasticsearch/Cassandra | | Visualization | Dashboards, exploration, correlation | Grafana, Kibana, Datadog UI | Read replicas; CDN for static assets | | Alerting | Evaluates rules, routes notifications | Alertmanager, Grafana Alerting, PagerDuty | HA pairs with deduplication | | Buffer/Queue | Decouples producers from consumers | Apache Kafka, Redis Streams, Amazon Kinesis | Kafka: partition-per-topic scaling | | Service Mesh Telemetry | Auto-instruments inter-service traffic | Istio, Linkerd, Envoy (built-in) | Sidecar proxy per pod | | Pipeline Orchestrator | Unified telemetry routing and processing | OpenTelemetry Collector, Vector | Gateway mode for centralized processing | ## Decision Tree > Full script: [decision-tree.txt](scripts/decision-tree.txt) (33 lines) ``` START: Choose your logging backend ├── Budget < $500/month AND log volume < 50 GB/day? │ ├── YES → Grafana Loki + Promtail (low resource, label-indexed) │ └── NO ↓ ├── Need full-text search across all log fields? # ... (see full script) ``` ## Step-by-Step Guide ### 1. Define your telemetry signals and data model Establish which of the three pillars — logs, metrics, traces — you need from day one. Most production systems need all three. Define a consistent naming convention for metrics (`service_name_operation_unit`), log fields (`timestamp`, `level`, `service`, `trace_id`, `message`), and trace attributes (`service.name`, `deployment.environment`). [src1] ```yaml # OpenTelemetry resource attributes (define once per service) resource: attributes: service.name: "payment-service" service.version: "2.1.0" deployment.environment: "production" service.namespace: "checkout" ``` **Verify**: All services emit `service.name` and `deployment.environment` in every telemetry signal. ### 2. Instrument applications with OpenTelemetry SDKs Add the OpenTelemetry SDK to each service. Use auto-instrumentation for common frameworks (Express, Flask, Spring Boot) and add manual spans for business-critical paths. [src1] ```bash # Python: install OpenTelemetry packages pip install opentelemetry-api==1.29.0 \ opentelemetry-sdk==1.29.0 \ opentelemetry-exporter-otlp==1.29.0 \ opentelemetry-instrumentation-flask==0.50b0 \ opentelemetry-instrumentation-requests==0.50b0 # Node.js: install OpenTelemetry packages npm install @opentelemetry/api@1.9.0 \ @opentelemetry/sdk-node@0.57.0 \ @opentelemetry/auto-instrumentations-node@0.55.0 \ @opentelemetry/exporter-trace-otlp-http@0.57.0 ``` **Verify**: `curl http://localhost:4318/v1/traces` returns 200 from the local OTel Collector. ### 3. Deploy the OpenTelemetry Collector as the central pipeline The OTel Collector acts as a vendor-neutral proxy that receives telemetry from all services, processes it (batching, sampling, enrichment), and exports to your chosen backends. Deploy as a DaemonSet in Kubernetes or a sidecar/gateway in VM environments. [src6] ```yaml # otel-collector-config.yaml receivers: otlp: protocols: grpc: endpoint: "0.0.0.0:4317" http: endpoint: "0.0.0.0:4318" processors: batch: timeout: 5s send_batch_size: 1024 memory_limiter: check_interval: 1s limit_mib: 512 spike_limit_mib: 128 attributes: actions: - key: environment value: production action: upsert exporters: otlphttp/loki: endpoint: "http://loki:3100/otlp" prometheusremotewrite: endpoint: "http://mimir:9009/api/v1/push" otlp/tempo: endpoint: "tempo:4317" tls: insecure: true service: pipelines: logs: receivers: [otlp] processors: [memory_limiter, batch] exporters: [otlphttp/loki] metrics: receivers: [otlp] processors: [memory_limiter, batch] exporters: [prometheusremotewrite] traces: receivers: [otlp] processors: [memory_limiter, batch] exporters: [otlp/tempo] ``` **Verify**: `curl -s http://localhost:13133/` returns `{"status":"Server available"}` from the Collector health check. ### 4. Set up log storage backend Choose between Elasticsearch (full-text search), Loki (label-indexed, low cost), or a managed service. Configure index lifecycle management (ILM) to auto-rotate and delete old indices. [src3] [src4] ```yaml # Elasticsearch ILM policy example PUT _ilm/policy/logs-policy { "policy": { "phases": { "hot": { "actions": { "rollover": { "max_size": "50gb", "max_age": "1d" } } }, "warm": { "min_age": "7d", "actions": { "shrink": { "number_of_shards": 1 }, "forcemerge": { "max_num_segments": 1 } } }, "cold": { "min_age": "30d", "actions": { "searchable_snapshot": { "snapshot_repository": "s3-repo" } } }, "delete": { "min_age": "90d", "actions": { "delete": {} } } } } } ``` **Verify**: `curl -s http://localhost:9200/_ilm/policy/logs-policy | jq .` shows the policy is active. ### 5. Configure metrics collection with Prometheus Deploy Prometheus with service discovery for your environment (Kubernetes annotations, Consul, or file-based targets). Define recording rules for pre-aggregation and alerting rules for SLO-based alerts. [src2] ```yaml # prometheus.yml global: scrape_interval: 15s evaluation_interval: 15s rule_files: - "recording_rules.yml" - "alerting_rules.yml" scrape_configs: - job_name: "kubernetes-pods" kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port] action: replace target_label: __address__ regex: (.+) replacement: ${1}:${2} alerting: alertmanagers: - static_configs: - targets: ["alertmanager:9093"] ``` **Verify**: `curl http://localhost:9090/api/v1/targets` → shows all expected targets as `"health": "up"`. ### 6. Build Grafana dashboards for visualization and alerting Create dashboards following the RED method (Rate, Errors, Duration) for services and USE method (Utilization, Saturation, Errors) for infrastructure. Set up alert rules with meaningful thresholds and route to appropriate channels. [src2] [src5] ```bash # Deploy Grafana with provisioned datasources docker run -d --name grafana \ -p 3000:3000 \ -e GF_SECURITY_ADMIN_PASSWORD=changeme \ -v ./grafana/provisioning:/etc/grafana/provisioning \ grafana/grafana:11.4.0 # Verify datasource connectivity curl -u admin:changeme \ http://localhost:3000/api/datasources/proxy/1/api/v1/query?query=up ``` **Verify**: Grafana UI at `http://localhost:3000` shows data sources as "connected" (green). ## Code Examples ### Python: Structured Logging with OpenTelemetry Context > Full script: [python-structured-logging-with-opentelemetry-conte.py](scripts/python-structured-logging-with-opentelemetry-conte.py) (33 lines) ```python # Input: Application events during request handling # Output: JSON log lines with trace_id, span_id, and structured fields import structlog # structlog==24.4.0 from opentelemetry import trace # Configure structlog for JSON output with OTel context # ... (see full script) ``` ### Node.js: Structured Logging with Pino and OpenTelemetry > Full script: [node-js-structured-logging-with-pino-and-opentelem.js](scripts/node-js-structured-logging-with-pino-and-opentelem.js) (31 lines) ```javascript // Input: HTTP requests to an Express service // Output: JSON log lines with trace context, request metadata const pino = require("pino"); // pino@9.6.0 const { trace } = require("@opentelemetry/api"); // @opentelemetry/api@1.9.0 const logger = pino({ # ... (see full script) ``` ### Python: Prometheus Metrics with Labels > Full script: [python-prometheus-metrics-with-labels.py](scripts/python-prometheus-metrics-with-labels.py) (33 lines) ```python # Input: HTTP request handling in a Flask/FastAPI service # Output: Prometheus metrics exposed at /metrics endpoint from prometheus_client import ( # prometheus-client==0.21.1 Counter, Histogram, Gauge, start_http_server ) # ... (see full script) ``` ### YAML: Complete Docker Compose Observability Stack > Full script: [yaml-complete-docker-compose-observability-stack.yml](scripts/yaml-complete-docker-compose-observability-stack.yml) (48 lines) ```yaml # Input: Docker environment needing full observability # Output: Running Loki + Prometheus + Tempo + Grafana + OTel Collector version: "3.9" services: otel-collector: # ... (see full script) ``` ## Anti-Patterns ### Wrong: Unstructured string concatenation in logs ```python # BAD — unstructured text makes automated parsing impossible import logging logger = logging.getLogger(__name__) def process_payment(user_id, amount): logger.info("Processing payment for user " + user_id + " amount: $" + str(amount)) # Output: "Processing payment for user u-123 amount: $49.99" # Cannot query by user_id or amount without regex parsing ``` ### Correct: Structured JSON logging with typed fields ```python # GOOD — structured fields are indexable, queryable, and alertable import structlog logger = structlog.get_logger() def process_payment(user_id: str, amount: float): logger.info("payment_processing", user_id=user_id, amount=amount, currency="USD") # Output: {"event":"payment_processing","user_id":"u-123","amount":49.99,"currency":"USD"} # Every field is independently queryable ``` ### Wrong: Logging sensitive data without redaction ```javascript // BAD — PII and secrets in plain text logs logger.info("User login", { email: user.email, // PII password: req.body.password, // credential token: session.jwt, // secret ssn: user.socialSecurity, // regulated data }); ``` ### Correct: Redacting sensitive fields before logging ```javascript // GOOD — redact sensitive fields, log only safe identifiers const pino = require("pino"); const logger = pino({ redact: ["password", "token", "ssn", "*.authorization"], }); logger.info("User login", { user_id: user.id, // safe identifier email: "[REDACTED]", // or omit entirely ip: req.ip, // may need consent in EU login_method: "password", }); ``` ### Wrong: Using a single high-cardinality metric label ```python # BAD — user_id as a Prometheus label creates millions of time series # This will crash Prometheus or cause massive memory usage REQUEST_COUNT = Counter( "http_requests_total", "Requests", ["method", "endpoint", "user_id"], # user_id = cardinality bomb ) ``` ### Correct: Keeping label cardinality bounded ```python # GOOD — use only bounded-cardinality labels; track per-user in logs/traces REQUEST_COUNT = Counter( "http_requests_total", "Requests", ["method", "endpoint", "status"], # all bounded enums ) # For per-user analytics, use log fields or trace attributes instead logger.info("request_completed", user_id=user_id, duration_ms=42) ``` ### Wrong: Sampling traces at 100% in production ```yaml # BAD — storing every single trace in production is extremely expensive # A service handling 10K req/s generates ~864M spans/day processors: # No sampling configured = 100% of traces stored ``` ### Correct: Tail-based sampling for intelligent trace retention ```yaml # GOOD — sample based on error status, latency, and a baseline rate processors: tail_sampling: decision_wait: 10s policies: - name: errors-always type: status_code status_code: { status_codes: [ERROR] } - name: slow-requests type: latency latency: { threshold_ms: 1000 } - name: baseline-sample type: probabilistic probabilistic: { sampling_percentage: 5 } ``` ## Common Pitfalls - **High-cardinality label explosion**: Using unbounded values (user IDs, request IDs, URLs with query params) as Prometheus labels causes memory exhaustion. Fix: keep label values to bounded enums; use `relabel_configs` to drop or hash high-cardinality labels. [src2] - **No log-level gating in production**: Emitting DEBUG logs for all services in production generates 5-10x more data, spiking storage costs. Fix: set default to `WARN` or `INFO` per service; use dynamic log-level adjustment via feature flags or config reload. [src7] - **Missing correlation IDs across signals**: Logs, metrics, and traces without shared identifiers (trace_id) are impossible to correlate during incidents. Fix: propagate `trace_id` via OpenTelemetry context; inject into all structured log fields automatically. [src1] - **Single-node Elasticsearch with no ILM**: Running Elasticsearch on a single node with default settings leads to disk exhaustion within weeks. Fix: configure index lifecycle management (ILM) with hot/warm/cold/delete phases from day one. [src3] - **Alerting on raw metrics instead of SLOs**: Alerts on CPU > 80% or error count > 100 produce constant false positives. Fix: define SLI/SLO-based alerts (e.g., error budget burn rate > 2x) that reflect actual user impact. [src5] - **No buffer between producers and consumers**: A direct Filebeat-to-Elasticsearch pipeline means Elasticsearch downtime causes log loss. Fix: insert Kafka or Redis Streams as a buffer to decouple ingestion from storage. [src3] - **Ignoring log rotation and retention**: Unbounded log storage fills disks, causes outages, and violates compliance policies. Fix: configure logrotate locally, ILM in Elasticsearch, and retention rules in Loki (`retention_period: 720h`). [src4] - **Alert fatigue from noisy thresholds**: Too many low-priority alerts desensitize the on-call team. Fix: classify alerts by severity (critical/warning/info), route only critical to PagerDuty, aggregate warnings into daily digests. [src7] ## Diagnostic Commands ```bash # Check OpenTelemetry Collector health curl -s http://localhost:13133/ | jq . # Verify Prometheus targets are being scraped curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}' # Check Elasticsearch cluster health curl -s http://localhost:9200/_cluster/health | jq '{status, number_of_nodes, active_shards}' # Verify Loki is receiving logs curl -s http://localhost:3100/ready # Query recent logs from Loki via LogQL curl -G -s http://localhost:3100/loki/api/v1/query_range \ --data-urlencode 'query={service_name="payment-service"} |= "error"' \ --data-urlencode 'limit=10' | jq . # Check Grafana datasource connectivity curl -u admin:changeme -s http://localhost:3000/api/datasources | jq '.[].name' # Verify Tempo is receiving traces curl -s http://localhost:3200/ready # Check Prometheus storage TSDB stats curl -s http://localhost:9090/api/v1/status/tsdb | jq '{headChunks: .data.headStats.numChunks, seriesCount: .data.headStats.numSeries}' # Monitor OTel Collector pipeline metrics curl -s http://localhost:8888/metrics | grep otelcol_receiver_accepted ``` ## Version History & Compatibility | Component | Current Version | Status | Key Change | |---|---|---|---| | OpenTelemetry Collector | 0.96.x (2026) | Stable (logs GA since 2024-04) | Logs signal reached GA; unified pipeline for all three pillars | | Prometheus | 3.x (2025+) | Current | Native OTLP ingestion, UTF-8 metric names, new UI | | Elasticsearch | 8.x (2022+) | Current | License changed to SSPL+Elastic; Lucene 9; vector search | | Grafana Loki | 3.x (2024+) | Current | OTLP native ingestion; structured metadata; bloom filters | | Grafana Tempo | 2.x (2023+) | Current | TraceQL query language; vParquet4 format | | Grafana | 11.x (2024+) | Current | Unified alerting; Explore Logs/Traces/Metrics | ## When to Use / When Not to Use | Use When | Don't Use When | Use Instead | |---|---|---| | Building microservices with >3 services that interact | Single monolith with <1K req/day | Simple file-based logging with logrotate | | Need to correlate events across services during incidents | Static websites or JAMstack with no backend | CDN analytics (Cloudflare, Vercel) | | Compliance requires audit logs with retention guarantees | Prototyping or hackathon with no uptime SLA | Console.log with local files | | Operating Kubernetes clusters in production | Running a single Docker container locally | Docker logs command | | Need SLO-based alerting with error budget tracking | Team has no on-call rotation or incident process | Simple uptime monitoring (Uptime Robot, Pingdom) | ## Important Caveats - OpenTelemetry SDK auto-instrumentation adds 1-5% latency overhead per service; benchmark before enabling in latency-sensitive hot paths - Elasticsearch 8.x uses the SSPL license (not Apache 2.0) — verify legal compliance for your use case; OpenSearch is the Apache 2.0 fork - Prometheus is pull-based by default — for short-lived jobs (serverless, batch), use the Pushgateway or switch to OTel push-based collection - Loki does not index log content (only labels) — complex full-text searches across log bodies are significantly slower than Elasticsearch - Managed observability services (Datadog, New Relic, Splunk) cost 3-10x more than self-hosted at scale but eliminate operational overhead — break-even is typically at 50-200 GB/day depending on team size - Trace sampling decisions must be made carefully: 100% sampling is prohibitively expensive, but too-aggressive sampling will miss rare errors ## Related Units - [APM Tool Comparison](/software/system-design/apm-comparison/2026) - [Distributed Tracing Architecture](/software/system-design/distributed-tracing/2026) - [Incident Response and Debugging](/software/system-design/incident-response/2026) - [Alerting and On-Call Rotation Design](/software/system-design/alerting-oncall/2026)