Docker Compose ELK Stack 8.x: Elasticsearch, Logstash, Kibana

TL;DR

• Bottom line: A production-ready ELK stack on Docker Compose requires pinned 8.x image versions, a setup container for user/password initialization, persistent volumes for data and certs, and vm.max_map_count=262144 on the host.
• Key tool/command: docker compose up -d with a properly configured docker-compose.yml pinning all services to the same 8.17.0 (or latest 8.x) tag.
• Watch out for: Elasticsearch 8.x enables security by default -- forgetting to set ELASTIC_PASSWORD and bootstrap user passwords causes startup failures and locked-out clusters.
• Works with: Docker Engine 20.10+, Docker Compose v2.0+, Linux/macOS/Windows (WSL2). All Elastic images from docker.elastic.co.

Constraints

• Set vm.max_map_count=262144 on the Docker host before starting Elasticsearch -- container will crash without it
• Pin all three images (ES, Logstash, Kibana) to the SAME version tag -- mixing versions causes compatibility errors
• NEVER use the latest Docker tag -- Elastic does not publish a latest tag
• Set ES_JAVA_OPTS -Xms and -Xmx to equal values, max 50% of available RAM, never exceeding 31 GB
• Do NOT use the logstash_system user for pipeline output -- it lacks index write permissions; create a dedicated logstash_internal user
• Elasticsearch 8.x enables security by default -- you MUST set ELASTIC_PASSWORD and configure user passwords

Quick Reference

Service Configuration Summary

Environment File (.env)

Decision Tree

START: What is your deployment scenario?
├── Development/testing on a single machine?
│   ├── YES → Use single-node mode (discovery.type=single-node)
│   │   ├── Need quick prototyping?
│   │   │   ├── YES → Disable security (xpack.security.enabled=false) -- DEV ONLY
│   │   │   └── NO → Keep security on, set ELASTIC_PASSWORD
│   └── NO (production) →
│       ├── Data volume < 100 GB/day?
│       │   ├── YES → Single-node ES is sufficient, enable security + TLS
│       │   └── NO → Multi-node cluster (3+ ES nodes)
│       │       ├── Need high availability?
│       │       │   ├── YES → 3 master-eligible + 2 data nodes minimum
│       │       │   └── NO → 3 combined master+data nodes
│       └── Need TLS between all components?
│           ├── YES → Use setup container with elasticsearch-certutil (see Step 2)
│           └── NO → Basic authentication only (passwords, no TLS)

Step-by-Step Guide

1. Set host system requirements

Elasticsearch requires elevated mmap limits. This MUST be done on the Docker host, not inside the container. [src2]

# Linux: set vm.max_map_count (required for Elasticsearch)
sudo sysctl -w vm.max_map_count=262144

# Make persistent across reboots
echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf

# Windows (WSL2): run in PowerShell as admin
wsl -d docker-desktop sh -c "sysctl -w vm.max_map_count=262144"

Verify: sysctl vm.max_map_count → expected: vm.max_map_count = 262144

2. Create project directory and environment file

Set up the directory structure with configuration files for each service. [src4]

mkdir -p docker-elk/{elasticsearch,logstash/pipeline,logstash/config,kibana/config}

cat > docker-elk/.env << 'EOF'
ELASTIC_VERSION=8.17.0
ELASTIC_PASSWORD=changeme
KIBANA_SYSTEM_PASSWORD=changeme
LOGSTASH_INTERNAL_PASSWORD=changeme
ES_MEM_LIMIT=1073741824
KB_MEM_LIMIT=1073741824
LS_MEM_LIMIT=1073741824
CLUSTER_NAME=docker-elk
LICENSE=basic
EOF

Verify: cat docker-elk/.env → all variables should be set

3. Create the docker-compose.yml

This is the core configuration file that defines all ELK services. [src1]

Full script: docker-compose.yml (89 lines)

Verify: docker compose config → validates the Compose file without starting services

4. Create Logstash pipeline configuration

The Logstash pipeline defines input sources, processing filters, and the Elasticsearch output. [src3]

Full script: logstash.conf (44 lines)

Verify: docker compose exec logstash logstash --config.test_and_exit → validates pipeline syntax

5. Create Kibana configuration

Configure Kibana to connect to Elasticsearch with authentication. [src5]

# kibana/config/kibana.yml
server.name: kibana
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://elasticsearch:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "${KIBANA_SYSTEM_PASSWORD}"
monitoring.ui.container.elasticsearch.enabled: true

Verify: curl -s http://localhost:5601/api/status | jq .status.overall.level → "available"

6. Start the stack and initialize users

Bootstrap the stack by starting Elasticsearch first, then initializing service account passwords. [src4]

cd docker-elk
docker compose up -d

# Wait for Elasticsearch to be healthy
until curl -s -u elastic:changeme http://localhost:9200/_cluster/health | grep -q '"status"'; do
  echo "Waiting for Elasticsearch..."; sleep 5
done

# Set kibana_system password
curl -s -X POST -u elastic:changeme \
  http://localhost:9200/_security/user/kibana_system/_password \
  -H "Content-Type: application/json" \
  -d '{"password":"changeme"}'

Verify: docker compose ps → all containers should show "healthy" or "running"

Code Examples

Docker Compose: Complete Single-Node ELK Stack

Full script: docker-compose.yml (89 lines)

# docker-compose.yml -- ELK Stack 8.x single-node development
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION}
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - xpack.security.enabled=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - es-data:/usr/share/elasticsearch/data
    ports:
      - "127.0.0.1:9200:9200"
      - "127.0.0.1:9300:9300"

Logstash Pipeline: JSON + Syslog + Beats Input

Full script: logstash.conf (44 lines)

# logstash/pipeline/logstash.conf
input {
  beats { port => 5044 }
  tcp { port => 5000; codec => json_lines }
  syslog { port => 5140 }
}

Elasticsearch Index Template: Log Data

curl -s -X PUT -u elastic:changeme \
  http://localhost:9200/_index_template/logs-template \
  -H "Content-Type: application/json" \
  -d '{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "index.lifecycle.name": "logs-policy"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "message": { "type": "text" },
        "level": { "type": "keyword" },
        "service": { "type": "keyword" }
      }
    }
  }
}'

ILM Policy: Log Retention

curl -s -X PUT -u elastic:changeme \
  http://localhost:9200/_ilm/policy/logs-policy \
  -H "Content-Type: application/json" \
  -d '{
  "policy": {
    "phases": {
      "hot": { "actions": { "rollover": { "max_primary_shard_size": "50gb", "max_age": "30d" } } },
      "warm": { "min_age": "30d", "actions": { "shrink": { "number_of_shards": 1 } } },
      "delete": { "min_age": "90d", "actions": { "delete": {} } }
    }
  }
}'

Anti-Patterns

Wrong: Running without persistent volumes

# BAD -- data lost when container restarts
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.17.0
    # No volumes defined -- all indices lost on restart

Correct: Named volumes for data persistence

# GOOD -- data survives container restarts and upgrades
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.17.0
    volumes:
      - es-data:/usr/share/elasticsearch/data
volumes:
  es-data:
    driver: local

Wrong: Using logstash_system for pipeline output

# BAD -- logstash_system is monitoring-only, cannot write indices
output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    user => "logstash_system"
    password => "${LOGSTASH_PASSWORD}"
  }
}

Correct: Dedicated logstash_internal user with write role

# GOOD -- logstash_internal has logstash_writer role with index write
output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    user => "logstash_internal"
    password => "${LOGSTASH_INTERNAL_PASSWORD}"
    index => "logstash-%{+YYYY.MM.dd}"
  }
}

Wrong: Mismatched image versions

# BAD -- mixing 8.17.0 and 8.15.0 causes version incompatibility
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.17.0
  logstash:
    image: docker.elastic.co/logstash/logstash:8.15.0
  kibana:
    image: docker.elastic.co/kibana/kibana:8.16.0

Correct: Single version variable for all images

# GOOD -- all services pinned via .env variable
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION}
  logstash:
    image: docker.elastic.co/logstash/logstash:${ELASTIC_VERSION}
  kibana:
    image: docker.elastic.co/kibana/kibana:${ELASTIC_VERSION}
# .env: ELASTIC_VERSION=8.17.0

Wrong: Unequal JVM heap sizes

# BAD -- Xms != Xmx causes heap resizing overhead and GC pauses
environment:
  - "ES_JAVA_OPTS=-Xms256m -Xmx2g"

Correct: Equal min and max heap

# GOOD -- equal Xms/Xmx eliminates heap resizing, 50% of container memory
environment:
  - "ES_JAVA_OPTS=-Xms1g -Xmx1g"

Common Pitfalls

• vm.max_map_count not set: Elasticsearch crashes on startup with max virtual memory areas vm.max_map_count [65530] is too low. Fix: sudo sysctl -w vm.max_map_count=262144 on the Docker host. [src2]
• Elasticsearch OOM with default memory: Docker's default memory limit is too low. Fix: Set mem_limit to at least 1 GB and ES_JAVA_OPTS to 50% of that. [src2]
• Kibana can't connect after security enabled: kibana_system password not set. Fix: Use _security/user/kibana_system/_password API after ES starts. [src4]
• Logstash pipeline fails silently: Output user lacks index write permissions. Fix: Create logstash_writer role with write, create_index, and manage privileges. [src3]
• Port 9200 exposed to the internet: Docker publishes to 0.0.0.0 by default. Fix: Bind to localhost with 127.0.0.1:9200:9200. [src1]
• Data directory permission errors: Elasticsearch can't write to /usr/share/elasticsearch/data. Fix: Ensure volume directory is owned by UID 1000 (chown -R 1000:1000 ./es-data). [src2]
• Logstash config not reloading: Changes require restart. Fix: Enable auto-reload with config.reload.automatic: true in logstash.yml. [src3]
• Kibana "server is not ready yet": Kibana takes 1-3 minutes to initialize. Fix: Add depends_on with health check on Elasticsearch and wait. [src6]

Diagnostic Commands

# Check Elasticsearch cluster health
curl -s -u elastic:changeme http://localhost:9200/_cluster/health?pretty

# Check Elasticsearch node stats (memory, disk, CPU)
curl -s -u elastic:changeme http://localhost:9200/_nodes/stats?pretty | jq '.nodes[].os'

# Check Logstash pipeline status
curl -s http://localhost:9600/_node/stats/pipelines?pretty

# Verify Logstash can reach Elasticsearch
docker compose exec logstash curl -s -u logstash_internal:changeme http://elasticsearch:9200

# Check Kibana status
curl -s http://localhost:5601/api/status | jq '.status.overall'

# View Elasticsearch container logs
docker compose logs elasticsearch --tail=50

# View Logstash pipeline errors
docker compose logs logstash --tail=50 | grep -i error

# Check vm.max_map_count on the host
sysctl vm.max_map_count

# List all Elasticsearch indices
curl -s -u elastic:changeme http://localhost:9200/_cat/indices?v

# Test Logstash TCP input
echo '{"message":"test log","level":"info"}' | nc localhost 5000

# Check Elasticsearch disk usage
curl -s -u elastic:changeme http://localhost:9200/_cat/allocation?v

Version History & Compatibility

When to Use / When Not to Use

Important Caveats

• Elasticsearch 8.x generates self-signed TLS certificates on first start -- for production, replace with proper CA-signed certificates or use the elasticsearch-certutil tool in a setup container
• The elastic superuser should NOT be used in production applications -- create dedicated users with minimum required privileges
• Docker Compose restarts may cause Elasticsearch cluster UUID conflicts if data volumes are corrupted -- always use docker compose down (not kill) for clean shutdown
• Logstash JVM heap is separate from Elasticsearch heap -- monitor both independently; Logstash typically needs 256 MB-1 GB depending on pipeline complexity
• On Docker Desktop for Mac/Windows, Elasticsearch performance is significantly lower than native Linux due to the VM layer -- expect 30-50% throughput reduction
• Index Lifecycle Management (ILM) policies should be configured after initial setup to prevent unbounded disk growth -- the default is no automatic cleanup