---
# === IDENTITY ===
id: software/system-design/analytics-metrics-pipeline/2026
canonical_question: "How do I design an analytics and metrics pipeline?"
aliases:
  - "analytics pipeline architecture"
  - "metrics pipeline design"
  - "real-time analytics pipeline"
  - "event streaming analytics architecture"
  - "Kafka Flink ClickHouse pipeline"
  - "observability metrics pipeline"
  - "data analytics pipeline system design"
  - "lambda vs kappa architecture"
entity_type: software_reference
domain: software > system-design > analytics_metrics_pipeline
region: global
jurisdiction: global
temporal_scope: 2020-2026

# === VERIFICATION ===
last_verified: 2026-02-23
confidence: 0.91
version: 1.0
first_published: 2026-02-23

# === TEMPORAL VALIDITY ===
temporal_validity:
  status: stable
  last_breaking_change: "Kafka 4.0 removed ZooKeeper dependency (KRaft-only, 2025-03); Flink 2.0 unified DataStream/Table API (2024-12); ClickHouse 24.x introduced Kafka table engine improvements (2024)"
  next_review: 2026-08-22
  change_sensitivity: medium

# === CONSTRAINTS ===
constraints:
  - "Never ingest raw, unbounded-cardinality labels (user IDs, session IDs) into a metrics store — this explodes storage and kills query performance"
  - "Always deploy a dead-letter queue (DLQ) for failed events — silently dropping messages causes invisible data loss"
  - "Kafka topic partition count cannot be decreased after creation — plan partition strategy before deployment"
  - "ClickHouse ReplacingMergeTree deduplication is eventual, not immediate — queries during merge may return duplicates; use FINAL keyword or design for idempotency"
  - "Stream processing state (Flink checkpoints, Kafka offsets) must be persisted to durable storage — in-memory-only state is lost on restart"

# === SKIP CONDITIONS ===
skip_this_unit_if:
  - condition: "User needs a simple dashboard for <100 req/s with no real-time requirement"
    use_instead: "Use a managed service like Datadog, New Relic, or Grafana Cloud — custom pipelines add unnecessary complexity at small scale"
  - condition: "User wants application performance monitoring (APM) tracing, not metrics aggregation"
    use_instead: "Evaluate OpenTelemetry + Jaeger/Tempo for distributed tracing"
  - condition: "User needs a data warehouse/lakehouse for batch analytics only"
    use_instead: "Evaluate Snowflake, BigQuery, or Databricks — stream processing adds complexity when batch is sufficient"

# === AGENT HINTS ===
inputs_needed:
  - key: event_volume
    question: "What is your expected event ingestion rate?"
    type: choice
    options: ["<10K events/sec", "10K-100K events/sec", "100K-1M events/sec", ">1M events/sec"]
  - key: latency_requirement
    question: "What query latency do you need for dashboards and alerts?"
    type: choice
    options: ["Sub-second (real-time)", "Seconds (near real-time)", "Minutes (micro-batch)", "Hours (batch OK)"]
  - key: processing_complexity
    question: "What kind of processing do you need on the event stream?"
    type: choice
    options: ["Simple aggregation (count, sum, avg)", "Windowed analytics (sliding/tumbling)", "Complex event processing (joins, patterns)", "ML inference on stream"]

# === DISTRIBUTION ===
canonical_source: "https://knowledgelib.io/software/system-design/analytics-metrics-pipeline/2026"
suggested_citation: "Source: knowledgelib.io — AI Knowledge Library (verified 2026-02-23)"

# === RELATED UNITS ===
related_kos:
  related_to:
    - id: "software/migrations/docker-compose-to-kubernetes/2026"
      label: "Docker Compose to Kubernetes (deploying pipeline infrastructure)"
    - id: "software/debugging/redis-memory-issues/2026"
      label: "Redis Memory Issues (caching layer troubleshooting)"
  solves:
    - id: "software/debugging/postgresql-slow-queries/2026"
      label: "PostgreSQL Slow Queries (offload analytics to OLAP store)"
  alternative_to:
    - id: "software/system-design/log-aggregation-pipeline/2026"
      label: "Log Aggregation Pipeline (ELK/Loki for logs vs metrics)"

# === SOURCES (7 authoritative sources) ===
sources:
  - id: src1
    title: "HTTP Analytics for 6M Requests per Second Using ClickHouse"
    author: Cloudflare Engineering
    url: https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/
    type: technical_blog
    published: 2023-05-18
    reliability: high
  - id: src2
    title: "Building Highly Reliable Data Pipelines at Datadog"
    author: Datadog Engineering
    url: https://www.datadoghq.com/blog/engineering/highly-reliable-data-pipelines/
    type: technical_blog
    published: 2023-09-14
    reliability: high
  - id: src3
    title: "Apache Flink — Use Cases: Event-Driven Applications, Stream & Batch Analytics"
    author: Apache Software Foundation
    url: https://flink.apache.org/what-is-flink/use-cases/
    type: official_docs
    published: 2024-01-15
    reliability: authoritative
  - id: src4
    title: "Lambda vs. Kappa Architecture — Choosing the Right Data Processing Architecture"
    author: nexocode
    url: https://nexocode.com/blog/posts/lambda-vs-kappa-architecture/
    type: technical_blog
    published: 2024-06-10
    reliability: moderate_high
  - id: src5
    title: "How We Use ClickHouse as a Real-Time Stream Processing Engine"
    author: Mux Engineering
    url: https://www.mux.com/blog/how-we-use-clickhouse-as-a-real-time-stream-processing-engine
    type: technical_blog
    published: 2024-03-22
    reliability: high
  - id: src6
    title: "Three Pesky Observability Anti-Patterns That Impact Developer Efficiency"
    author: Chronosphere
    url: https://chronosphere.io/learn/three-pesky-observability-anti-patterns-that-impact-developer-efficiency/
    type: technical_blog
    published: 2024-02-08
    reliability: moderate_high
  - id: src7
    title: "Data Pipeline Architecture: 9 Patterns & Best Practices for Scalable Systems"
    author: Alation
    url: https://www.alation.com/blog/data-pipeline-architecture-patterns/
    type: technical_blog
    published: 2024-11-20
    reliability: moderate_high
---

# How to Design an Analytics and Metrics Pipeline

## TL;DR

- **Bottom line**: An analytics/metrics pipeline consists of four layers — ingestion (collect events), processing (transform/aggregate), storage (OLAP-optimized), and serving (query/alert) — connected by a durable message bus like Kafka.
- **Key tool/command**: `Kafka (ingestion) → Flink (processing) → ClickHouse (storage) → Grafana (serving)` is the most battle-tested open-source stack for high-throughput metrics.
- **Watch out for**: Unbounded cardinality in metric labels — a single high-cardinality tag (like user_id) can multiply storage costs 100x and destroy query performance.
- **Works with**: Any cloud or on-premises environment; Kafka 3.x-4.x, Flink 1.18-2.x, ClickHouse 23.x-24.x, or managed equivalents (Confluent, Amazon MSK, Kinesis, BigQuery).

## Constraints
<!-- Agents: read this section before recommending any action from this unit.
     These are hard constraints that must not be violated. -->

- Never ingest raw unbounded-cardinality labels (user IDs, session IDs) into a metrics time-series store — aggregate or hash them first
- Always deploy a dead-letter queue (DLQ) for malformed or failed events — silent drops create invisible data loss that corrupts dashboards
- Kafka topic partition count cannot be decreased — over-partitioning wastes resources; under-partitioning limits parallelism. Plan partitions = 2x expected peak consumer parallelism
- ClickHouse MergeTree deduplication is eventual (happens during background merges) — design write paths to be idempotent or use `FINAL` in queries
- Stream processing checkpoints (Flink savepoints, Kafka consumer offsets) must be persisted to durable storage (S3, HDFS) — in-memory state is lost on failure

## Quick Reference

| Component | Role | Technology Options | Scaling Strategy |
|---|---|---|---|
| **Event Producers** | Emit structured events/metrics from apps | OpenTelemetry SDK, StatsD, custom emitters | Horizontal — add producers independently |
| **Message Bus** | Durable, ordered event transport | Apache Kafka, Amazon MSK, Redpanda, Pulsar | Partition-based — add partitions + brokers |
| **Schema Registry** | Enforce event schema contracts | Confluent Schema Registry, AWS Glue | Single cluster — scales with broker count |
| **Stream Processor** | Transform, aggregate, enrich events in-flight | Apache Flink, Kafka Streams, Spark Structured Streaming | Task-parallelism — add TaskManagers/executors |
| **Pre-Aggregation** | Reduce cardinality before storage | Flink windowed aggregation, ClickHouse Materialized Views | Tied to processor or storage scaling |
| **OLAP Storage** | Columnar analytics store for fast queries | ClickHouse, Apache Druid, Apache Pinot, TimescaleDB | Shard-based — add shards for write throughput, replicas for read |
| **Object Storage** | Raw event archive / data lake tier | S3, GCS, MinIO | Virtually unlimited — cost scales linearly |
| **Batch Processor** | Backfill, reprocess, train models on historical data | Apache Spark, dbt, Apache Airflow (orchestration) | Cluster auto-scaling — Spark/EMR elastic resize |
| **Query Engine** | SQL interface for analysts and dashboards | ClickHouse native, Trino/Presto, Grafana direct | Read replicas or caching layer |
| **Serving / Visualization** | Dashboards, alerts, API endpoints | Grafana, Superset, custom REST APIs | Horizontal — stateless query proxies |
| **Alerting** | Threshold and anomaly detection on metrics | Grafana Alerting, Prometheus Alertmanager, PagerDuty | Stateless evaluators — scale horizontally |
| **Dead-Letter Queue** | Capture failed/malformed events for reprocessing | Kafka DLQ topic, S3 error bucket | Same as message bus scaling |

## Decision Tree

```
START: What is your event volume and latency requirement?
├── <10K events/sec AND minutes-latency OK?
│   ├── YES → Simple stack: producers → Kafka → batch consumer → PostgreSQL/TimescaleDB → Grafana
│   │         (No stream processor needed — Kafka consumer writes directly to storage)
│   └── NO ↓
├── 10K-100K events/sec OR sub-second latency needed?
│   ├── YES → Standard streaming: Kafka → Flink (windowed aggregation) → ClickHouse → Grafana
│   │         (Kappa architecture — single stream processing path)
│   └── NO ↓
├── >100K events/sec OR complex event processing (joins, patterns)?
│   ├── YES → Full pipeline: Kafka → Flink (CEP + aggregation) → ClickHouse cluster (sharded) → Grafana
│   │         Add pre-aggregation in Flink to reduce ClickHouse write amplification
│   └── NO ↓
├── >1M events/sec (hyperscale)?
│   ├── YES → Multi-tier: Kafka (tiered storage) → Flink (multi-stage DAG) → ClickHouse Cloud/cluster
│   │         Add sampling at ingestion, rollup tables in ClickHouse, read replicas for queries
│   └── NO ↓
├── Need both real-time dashboards AND accurate historical backfill?
│   ├── YES → Lambda architecture: speed layer (Flink → ClickHouse) + batch layer (Spark → S3 → ClickHouse)
│   │         Merge results at query time. Higher operational cost but guarantees accuracy.
│   └── NO ↓
└── DEFAULT → Kappa architecture with Kafka + Flink + ClickHouse. Add batch reprocessing only if needed.
```

## Step-by-Step Guide

### 1. Define event schema and set up Schema Registry

Start with a well-defined event schema using Avro or Protobuf. Register it in a Schema Registry to enforce contracts between producers and consumers. Schema evolution (adding optional fields) is safe; removing or renaming fields requires a new schema version. [src2]

```protobuf
// metrics_event.proto — Protobuf schema for pipeline events
syntax = "proto3";
package analytics;

message MetricEvent {
  string event_id = 1;       // UUID for deduplication
  string metric_name = 2;    // e.g., "http_request_duration_ms"
  double value = 3;
  int64 timestamp_ms = 4;    // Unix epoch milliseconds
  map<string, string> tags = 5; // bounded-cardinality labels only
  string source = 6;         // producing service name
}
```

**Verify**: Check schema registered — `curl http://schema-registry:8081/subjects` → should list `metrics_event-value`

### 2. Deploy Kafka as the central message bus

Configure Kafka topics with appropriate partition counts and replication. Use `min.insync.replicas=2` with `acks=all` for durability. Partition by a low-cardinality key (metric_name, service_name) to ensure ordering within a metric while distributing load. [src1]

```bash
# Create metrics topic — 12 partitions, replication factor 3
kafka-topics.sh --create \
  --bootstrap-server kafka:9092 \
  --topic metrics-events \
  --partitions 12 \
  --replication-factor 3 \
  --config retention.ms=604800000 \
  --config min.insync.replicas=2 \
  --config cleanup.policy=delete

# Create DLQ topic for failed events
kafka-topics.sh --create \
  --bootstrap-server kafka:9092 \
  --topic metrics-events-dlq \
  --partitions 3 \
  --replication-factor 3
```

**Verify**: `kafka-topics.sh --describe --topic metrics-events` → should show 12 partitions, ISR=3

### 3. Implement stream processing with Flink

Deploy a Flink job that consumes from Kafka, applies windowed aggregation (e.g., 1-minute tumbling windows), and writes results to ClickHouse. Enable checkpointing to S3/HDFS for fault tolerance. [src3]

```java
// Flink pipeline: Kafka source → windowed aggregation → ClickHouse sink
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.enableCheckpointing(60_000, CheckpointingMode.EXACTLY_ONCE);
env.getCheckpointConfig().setCheckpointStorage("s3://checkpoints/metrics-pipeline");

KafkaSource<MetricEvent> source = KafkaSource.<MetricEvent>builder()
    .setBootstrapServers("kafka:9092")
    .setTopics("metrics-events")
    .setGroupId("metrics-processor")
    .setStartingOffsets(OffsetsInitializer.committedOffsets(OffsetResetStrategy.EARLIEST))
    .setDeserializer(new ProtoDeserializer<>(MetricEvent.class))
    .build();

DataStream<MetricEvent> events = env.fromSource(source, WatermarkStrategy
    .<MetricEvent>forBoundedOutOfOrderness(Duration.ofSeconds(5))
    .withTimestampAssigner((event, ts) -> event.getTimestampMs()), "kafka-source");

// 1-minute tumbling window aggregation
events
    .keyBy(MetricEvent::getMetricName)
    .window(TumblingEventTimeWindows.of(Time.minutes(1)))
    .aggregate(new MetricAggregator())  // sum, count, min, max, p50, p99
    .addSink(new ClickHouseSink("jdbc:clickhouse://clickhouse:8123/metrics"));

env.execute("metrics-pipeline");
```

**Verify**: Flink Web UI at `http://flink-jobmanager:8081` → job should show "RUNNING" with checkpoints succeeding

### 4. Configure ClickHouse storage with rollup tables

Create a raw events table using MergeTree and materialized views for pre-aggregated rollups. Use ClickHouse's built-in Kafka engine for direct ingestion if Flink is not needed, or write from Flink via JDBC. [src1] [src5]

```sql
-- Raw metrics table (receives from Flink or Kafka engine)
CREATE TABLE metrics_raw (
    event_id       String,
    metric_name    LowCardinality(String),
    value          Float64,
    timestamp_ms   DateTime64(3),
    tags           Map(LowCardinality(String), String),
    source         LowCardinality(String)
) ENGINE = ReplacingMergeTree(timestamp_ms)
PARTITION BY toYYYYMM(timestamp_ms)
ORDER BY (metric_name, source, timestamp_ms, event_id)
TTL timestamp_ms + INTERVAL 90 DAY
SETTINGS index_granularity = 8192;

-- 1-minute rollup materialized view
CREATE MATERIALIZED VIEW metrics_1m_mv TO metrics_1m AS
SELECT
    metric_name,
    source,
    toStartOfMinute(timestamp_ms) AS ts,
    count()                        AS cnt,
    sum(value)                     AS val_sum,
    min(value)                     AS val_min,
    max(value)                     AS val_max,
    avg(value)                     AS val_avg,
    quantile(0.50)(value)          AS val_p50,
    quantile(0.99)(value)          AS val_p99
FROM metrics_raw
GROUP BY metric_name, source, ts;

CREATE TABLE metrics_1m (
    metric_name    LowCardinality(String),
    source         LowCardinality(String),
    ts             DateTime,
    cnt            AggregateFunction(count, UInt64),
    val_sum        AggregateFunction(sum, Float64),
    val_min        AggregateFunction(min, Float64),
    val_max        AggregateFunction(max, Float64),
    val_avg        AggregateFunction(avg, Float64),
    val_p50        AggregateFunction(quantile(0.50), Float64),
    val_p99        AggregateFunction(quantile(0.99), Float64)
) ENGINE = AggregatingMergeTree()
PARTITION BY toYYYYMM(ts)
ORDER BY (metric_name, source, ts)
TTL ts + INTERVAL 365 DAY;
```

**Verify**: `SELECT count() FROM metrics_raw` → should increase as events flow; `SELECT count() FROM metrics_1m` → should populate within 1 minute

### 5. Set up serving layer with Grafana

Connect Grafana to ClickHouse using the official ClickHouse data source plugin. Create dashboards querying the rollup tables (not raw) for sub-second response. Configure alerts on metric thresholds. [src5]

```yaml
# grafana/provisioning/datasources/clickhouse.yml
apiVersion: 1
datasources:
  - name: ClickHouse-Metrics
    type: grafana-clickhouse-datasource
    url: http://clickhouse:8123
    jsonData:
      defaultDatabase: metrics
      protocol: http
    secureJsonData:
      password: "${CLICKHOUSE_PASSWORD}"
```

**Verify**: Grafana → Explore → select ClickHouse-Metrics → run `SELECT count() FROM metrics_1m WHERE ts > now() - INTERVAL 1 HOUR` → should return data

### 6. Deploy monitoring for the pipeline itself

Instrument every pipeline component. Kafka brokers expose JMX metrics; Flink exposes via Prometheus reporter; ClickHouse exposes system tables. Alert on consumer lag, checkpoint failures, and ClickHouse merge queue depth. [src2] [src6]

```bash
# Check Kafka consumer group lag
kafka-consumer-groups.sh --bootstrap-server kafka:9092 \
  --describe --group metrics-processor
# Look for LAG column — should be <1000 for healthy pipeline

# Flink metrics via Prometheus endpoint
curl http://flink-taskmanager:9249/metrics | grep checkpoint

# ClickHouse merge health
clickhouse-client -q "SELECT table, count() as merges, sum(rows_read) FROM system.merges GROUP BY table"
```

**Verify**: All three commands return data without errors; Kafka LAG < 1000; Flink checkpoints completing; ClickHouse merges progressing

## Code Examples

### Python: Kafka Producer with OpenTelemetry-Style Events

> Full script: [python-kafka-producer-with-opentelemetry-style-eve.py](scripts/python-kafka-producer-with-opentelemetry-style-eve.py) (28 lines)

```python
# Input:  Application metrics (HTTP request durations, counters)
# Output: Structured events published to Kafka topic
from confluent_kafka import Producer  # confluent-kafka==2.6.1
import json, uuid, time
producer = Producer({
# ... (see full script)
```

### SQL: ClickHouse Analytical Queries on Rollup Tables

> Full script: [sql-clickhouse-analytical-queries-on-rollup-tables.sql](scripts/sql-clickhouse-analytical-queries-on-rollup-tables.sql) (26 lines)

```sql
-- Input:  Pre-aggregated 1-minute rollup table
-- Output: Dashboard-ready time series with percentiles
-- Service latency p99 over last 24 hours (1-minute granularity)
SELECT
    ts,
# ... (see full script)
```

## Anti-Patterns

### Wrong: Storing high-cardinality fields as metric labels

```python
# BAD — user_id as a tag creates millions of unique time series
emit_metric("page_load_time_ms", 320.0, {
    "user_id": "u-839201",        # unbounded cardinality!
    "session_id": "s-28d7f3a",    # unbounded cardinality!
    "page": "/dashboard",
})
```

### Correct: Aggregate or move high-cardinality to event logs

```python
# GOOD — only bounded-cardinality tags in metrics
emit_metric("page_load_time_ms", 320.0, {
    "page": "/dashboard",          # bounded: ~100 pages
    "region": "us-east-1",         # bounded: ~20 regions
    "device_type": "mobile",       # bounded: 3 options
})
# High-cardinality data goes to a separate events/logs table
# with different storage and retention policies
```

### Wrong: No backpressure handling — fast producer overwhelms slow consumer

```python
# BAD — producer fires and forgets without flow control
for event in infinite_event_stream():
    producer.produce("metrics-events", value=serialize(event))
    # No flush, no error callback, no queue size check
    # Kafka internal buffer fills up → BufferError → data loss
```

### Correct: Implement backpressure with bounded buffers and error handling

```python
# GOOD — respect producer buffer limits and handle errors
MAX_QUEUE = 100_000
for event in infinite_event_stream():
    try:
        producer.produce("metrics-events", value=serialize(event),
                         callback=delivery_report)
        producer.poll(0)            # trigger callbacks, non-blocking
    except BufferError:
        producer.poll(1.0)          # backpressure: wait for buffer space
        producer.produce("metrics-events", value=serialize(event),
                         callback=delivery_report)
```

### Wrong: Single monolithic ClickHouse table without rollups

```sql
-- BAD — querying months of raw data for a dashboard panel
SELECT toStartOfMinute(timestamp_ms) AS ts, avg(value)
FROM metrics_raw
WHERE metric_name = 'cpu_usage'
  AND timestamp_ms > now() - INTERVAL 30 DAY  -- scans billions of rows
GROUP BY ts ORDER BY ts;
-- Result: 30+ second query, high CPU, blocks other queries
```

### Correct: Query pre-aggregated rollup tables

```sql
-- GOOD — query the 1-minute materialized view rollup
SELECT ts, avgMerge(val_avg) AS avg_cpu
FROM metrics_1m
WHERE metric_name = 'cpu_usage'
  AND ts > now() - INTERVAL 30 DAY  -- scans only rollup rows
GROUP BY ts ORDER BY ts;
-- Result: sub-second query, 100-1000x fewer rows scanned
```

## Common Pitfalls

- **Consumer lag spiraling**: Consumers fall behind producers, lag grows unbounded, alerts fire late. Fix: increase partition count and consumer instances in parallel; enable `auto.offset.reset=latest` for non-critical metrics to skip backlog. [src2]
- **Schema evolution breaking consumers**: Adding a required field in Avro/Protobuf breaks all downstream consumers. Fix: always use backward-compatible evolution — add optional fields only, never remove or rename. Use Schema Registry compatibility checks (`BACKWARD` mode). [src7]
- **ClickHouse merge pressure**: Too many small inserts cause thousands of parts, triggering "Too many parts" errors. Fix: batch inserts to ClickHouse in chunks of 10K-100K rows; use Kafka table engine or Buffer engine to absorb write bursts. [src1]
- **Missing watermarks in Flink**: Without watermarks, event-time windows never close on idle partitions. Fix: configure `withIdleness(Duration.ofMinutes(1))` on the watermark strategy. [src3]
- **No data retention policy**: Raw events accumulate forever, storage costs grow unbounded. Fix: set TTL on ClickHouse tables (`TTL timestamp_ms + INTERVAL 90 DAY`) and Kafka topic retention (`retention.ms=604800000` for 7 days). [src1]
- **Alerting on raw metrics instead of rollups**: Querying raw tables for alerts adds unpredictable latency to alert evaluation. Fix: alerts should query rollup/aggregated tables or use a dedicated alerting pipeline (Flink CEP → alert sink). [src6]
- **Single Kafka cluster for all environments**: Dev, staging, and prod sharing a single Kafka cluster leads to resource contention and accidental cross-environment data leaks. Fix: separate Kafka clusters per environment, or at minimum use ACLs + topic naming conventions. [src2]

## Diagnostic Commands

```bash
# Check Kafka cluster health and broker count
kafka-metadata.sh --snapshot /var/kafka-logs/__cluster_metadata-0/00000000000000000000.log --cluster-id

# Monitor consumer group lag (key health metric)
kafka-consumer-groups.sh --bootstrap-server kafka:9092 \
  --describe --group metrics-processor \
  | awk 'NR>1 {sum+=$6} END {print "Total lag:", sum}'

# Flink job status and checkpoint health
curl -s http://flink-jobmanager:8081/jobs | jq '.jobs[] | {id, status}'
curl -s http://flink-jobmanager:8081/jobs/<job-id>/checkpoints | jq '.latest'

# ClickHouse insert rate and merge health
clickhouse-client -q "SELECT event_time, query, written_rows FROM system.query_log WHERE type='QueryFinish' AND query LIKE 'INSERT%' ORDER BY event_time DESC LIMIT 10"
clickhouse-client -q "SELECT database, table, elapsed, progress FROM system.merges"

# ClickHouse table sizes and part counts
clickhouse-client -q "SELECT table, formatReadableSize(sum(bytes_on_disk)) AS size, count() AS parts FROM system.parts WHERE active GROUP BY table ORDER BY sum(bytes_on_disk) DESC"
```

## Version History & Compatibility

| Technology | Current Version | Key Changes | Migration Notes |
|---|---|---|---|
| Apache Kafka | 4.0 (2025) | ZooKeeper removed — KRaft-only mode | Migrate from ZK to KRaft before upgrading past 3.x |
| Apache Flink | 2.0 (2024) | Unified DataStream + Table API | Flink 1.x DataStream code still works but new Table API preferred |
| ClickHouse | 24.x (2024) | Improved Kafka table engine, new JSON type | Backward-compatible; new features opt-in |
| Kafka Streams | 3.8 (2024) | Improved state store backends | No breaking changes from 3.x |
| Redpanda | 24.x (2024) | Kafka API-compatible, no JVM | Drop-in Kafka replacement — same client libraries |
| Grafana | 11.x (2024) | ClickHouse plugin v4, unified alerting | Dashboard JSON forward-compatible |

## When to Use / When Not to Use

| Use When | Don't Use When | Use Instead |
|---|---|---|
| >10K events/sec needing sub-second dashboards | <1K events/sec with daily reporting | Managed APM (Datadog, New Relic) or simple PostgreSQL |
| Need custom aggregation logic (percentiles, funnels, cohorts) | Standard infra metrics (CPU, memory, disk) | Prometheus + Grafana (purpose-built for infra metrics) |
| Multi-source data (apps, infra, business events) unified in one store | Single-source application logs only | ELK stack or Grafana Loki for log aggregation |
| Need to replay/reprocess historical data with updated logic | One-time data migration or ETL job | Apache Spark batch job or dbt transformation |
| Cost-sensitive at scale (own infra cheaper than SaaS above 100TB/month) | Small team without dedicated data engineering | SaaS observability — operational cost outweighs infra savings |

## Important Caveats

- Kafka 4.0 removed ZooKeeper entirely — all new deployments must use KRaft mode. Existing ZooKeeper-based clusters must migrate before upgrading past 3.x. [src4]
- ClickHouse's `ReplacingMergeTree` and `AggregatingMergeTree` perform deduplication/aggregation during background merges, not at insert time — queries may see temporary duplicates. Use the `FINAL` modifier for guaranteed correctness at the cost of query speed. [src5]
- Flink exactly-once semantics require both source (Kafka) and sink (ClickHouse) to support transactions — ClickHouse does not support XA transactions, so use idempotent writes (dedup by event_id) instead. [src3]
- Lambda architecture (separate batch + stream paths) provides the strongest correctness guarantees but doubles operational complexity — prefer Kappa architecture (single stream path) unless you have a proven need for batch reprocessing accuracy. [src4]
- At hyperscale (>1M events/sec), network bandwidth between Kafka and ClickHouse often becomes the bottleneck before CPU or disk — enable `zstd` compression on both Kafka topics and ClickHouse columns. [src1]

## Related Units

- [Docker Compose to Kubernetes Migration](/software/migrations/docker-compose-to-kubernetes/2026) — deploying pipeline infrastructure on K8s
- [PostgreSQL Slow Queries](/software/debugging/postgresql-slow-queries/2026) — offload analytics workloads to OLAP store
- [Redis Memory Issues](/software/debugging/redis-memory-issues/2026) — caching layer for metrics serving