--- # === IDENTITY === id: software/devops/docker-elk-stack/2026 canonical_question: "Docker Compose reference: ELK Stack 8.x (Elasticsearch, Logstash, Kibana)" aliases: - "ELK stack Docker Compose setup" - "Elasticsearch Logstash Kibana docker-compose.yml" - "docker compose elk 8.x" - "elastic stack docker deployment" - "ELK Docker containers configuration" - "docker-compose Elasticsearch Kibana Logstash" - "how to run ELK stack in Docker" - "elk docker compose production setup" entity_type: software_reference domain: software > devops > Docker ELK Stack region: global jurisdiction: global temporal_scope: 2024-2026 # === VERIFICATION === last_verified: 2026-02-27 confidence: 0.93 version: 1.0 first_published: 2026-02-27 # === TEMPORAL VALIDITY === temporal_validity: status: stable last_breaking_change: "Elasticsearch 8.0 (Feb 2022): security enabled by default, TLS auto-configured, Basic/Trial license split" next_review: 2026-08-26 change_sensitivity: medium # === CONSTRAINTS === constraints: - "Set vm.max_map_count=262144 on the Docker host before starting Elasticsearch -- container will crash without it" - "Pin all three images (ES, Logstash, Kibana) to the SAME version tag -- mixing versions causes compatibility errors" - "NEVER use the 'latest' Docker tag -- Elastic does not publish a 'latest' tag" - "Set ES_JAVA_OPTS -Xms and -Xmx to equal values, max 50% of available RAM, never exceeding 31GB" - "Do NOT use the logstash_system user for pipeline output -- it lacks index write permissions; create a dedicated logstash_internal user" - "Elasticsearch 8.x enables security by default -- you MUST set ELASTIC_PASSWORD and configure user passwords" # === SKIP CONDITIONS === skip_this_unit_if: - condition: "Need a managed/cloud Elastic deployment, not self-hosted Docker" use_instead: "Elastic Cloud or AWS OpenSearch managed service documentation" - condition: "Using OpenSearch (AWS fork) instead of Elastic Stack" use_instead: "software/devops/docker-opensearch-stack/2026" - condition: "Need only Elasticsearch without Logstash/Kibana" use_instead: "software/devops/docker-elasticsearch/2026" # === AGENT HINTS === inputs_needed: - key: "deployment_mode" question: "Is this for development/testing or production use?" type: choice options: ["development", "production"] - key: "cluster_size" question: "Do you need a single-node or multi-node Elasticsearch cluster?" type: choice options: ["single-node", "multi-node (3 nodes)"] - key: "security_level" question: "Do you need TLS/SSL encryption between components?" type: choice options: ["basic (passwords only)", "full TLS (inter-node + HTTPS)"] # === DISTRIBUTION === canonical_source: "https://knowledgelib.io/software/devops/docker-elk-stack/2026" suggested_citation: "Source: knowledgelib.io -- AI Knowledge Library (verified 2026-02-27)" # === RELATED UNITS === related_kos: depends_on: [] related_to: - id: "software/devops/docker-compose-networking/2026" label: "Docker Compose Networking Guide" - id: "software/devops/docker-volumes-persistence/2026" label: "Docker Volumes & Data Persistence" solves: [] alternative_to: - id: "software/devops/docker-opensearch-stack/2026" label: "Docker OpenSearch Stack (AWS Fork)" often_confused_with: - id: "software/devops/docker-prometheus-grafana/2026" label: "Docker Prometheus + Grafana Stack (metrics, not logs)" # === SOURCES (7 authoritative sources) === sources: - id: src1 title: "Start a multi-node cluster with Docker Compose" author: Elastic url: https://www.elastic.co/docs/deploy-manage/deploy/self-managed/install-elasticsearch-docker-compose type: official_docs published: 2025-12-01 reliability: authoritative - id: src2 title: "Install Elasticsearch with Docker" author: Elastic url: https://www.elastic.co/guide/en/elasticsearch/reference/8.19/docker.html type: official_docs published: 2025-12-01 reliability: authoritative - id: src3 title: "Configuring Logstash for Docker" author: Elastic url: https://www.elastic.co/guide/en/logstash/8.19/docker-config.html type: official_docs published: 2025-12-01 reliability: authoritative - id: src4 title: "docker-elk: The Elastic stack powered by Docker and Compose" author: deviantony url: https://github.com/deviantony/docker-elk type: community_resource published: 2025-11-01 reliability: high - id: src5 title: "Getting started with the Elastic Stack and Docker-Compose" author: Elastic Blog url: https://www.elastic.co/blog/getting-started-with-the-elastic-stack-and-docker-compose type: technical_blog published: 2024-06-01 reliability: high - id: src6 title: "How to Set Up an ELK Stack with Docker Compose" author: OneUptime url: https://oneuptime.com/blog/post/2026-02-08-how-to-set-up-an-elk-stack-elasticsearch-logstash-kibana-with-docker-compose/view type: technical_blog published: 2026-02-08 reliability: moderate_high - id: src7 title: "Deploying ELK Stack with Docker Compose (2025 Edition)" author: IPNET url: https://ipnet.xyz/2025/06/deploying-elk-stack-with-docker-compose-2025-edition/ type: technical_blog published: 2025-06-01 reliability: moderate_high --- # Docker Compose ELK Stack 8.x: Elasticsearch, Logstash, Kibana ## TL;DR - **Bottom line**: A production-ready ELK stack on Docker Compose requires pinned 8.x image versions, a setup container for user/password initialization, persistent volumes for data and certs, and `vm.max_map_count=262144` on the host. - **Key tool/command**: `docker compose up -d` with a properly configured `docker-compose.yml` pinning all services to the same `8.17.0` (or latest 8.x) tag. - **Watch out for**: Elasticsearch 8.x enables security by default -- forgetting to set `ELASTIC_PASSWORD` and bootstrap user passwords causes startup failures and locked-out clusters. - **Works with**: Docker Engine 20.10+, Docker Compose v2.0+, Linux/macOS/Windows (WSL2). All Elastic images from `docker.elastic.co`. ## Constraints - Set `vm.max_map_count=262144` on the Docker host before starting Elasticsearch -- container will crash without it - Pin all three images (ES, Logstash, Kibana) to the SAME version tag -- mixing versions causes compatibility errors - NEVER use the `latest` Docker tag -- Elastic does not publish a `latest` tag - Set `ES_JAVA_OPTS` `-Xms` and `-Xmx` to equal values, max 50% of available RAM, never exceeding 31GB - Do NOT use the `logstash_system` user for pipeline output -- it lacks index write permissions; create a dedicated `logstash_internal` user - Elasticsearch 8.x enables security by default -- you MUST set `ELASTIC_PASSWORD` and configure user passwords ## Quick Reference | Service | Image | Ports | Volumes | Key Env | |---|---|---|---|---| | Elasticsearch | `docker.elastic.co/elasticsearch/elasticsearch:8.17.0` | `9200:9200` (API), `9300:9300` (transport) | `es-data:/usr/share/elasticsearch/data` | `discovery.type=single-node`, `ES_JAVA_OPTS=-Xms1g -Xmx1g`, `ELASTIC_PASSWORD`, `xpack.security.enabled=true` | | Logstash | `docker.elastic.co/logstash/logstash:8.17.0` | `5044:5044` (Beats), `5000:5000/tcp` (TCP input), `9600:9600` (monitoring API) | `./logstash/pipeline:/usr/share/logstash/pipeline:ro`, `./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro` | `LS_JAVA_OPTS=-Xms256m -Xmx256m`, `LOGSTASH_INTERNAL_PASSWORD` | | Kibana | `docker.elastic.co/kibana/kibana:8.17.0` | `5601:5601` | `./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:ro` | `KIBANA_SYSTEM_PASSWORD`, `ELASTICSEARCH_HOSTS=https://elasticsearch:9200` | | Setup (init) | `docker.elastic.co/elasticsearch/elasticsearch:8.17.0` | none | `es-certs:/usr/share/elasticsearch/config/certs` | `ELASTIC_PASSWORD`, `KIBANA_PASSWORD`, `LOGSTASH_PASSWORD` | **Environment File (.env):** | Variable | Default | Purpose | |---|---|---| | `ELASTIC_VERSION` | `8.17.0` | Stack version for all images | | `ELASTIC_PASSWORD` | `changeme` | Superuser password | | `KIBANA_SYSTEM_PASSWORD` | `changeme` | Kibana service account password | | `LOGSTASH_INTERNAL_PASSWORD` | `changeme` | Logstash pipeline output password | | `ES_MEM_LIMIT` | `1073741824` (1 GB) | Elasticsearch container memory limit | | `KB_MEM_LIMIT` | `1073741824` (1 GB) | Kibana container memory limit | | `LS_MEM_LIMIT` | `1073741824` (1 GB) | Logstash container memory limit | ## Decision Tree ``` START: What is your deployment scenario? |-- Development/testing on a single machine? | |-- YES --> Use single-node mode (discovery.type=single-node) | | |-- Need quick prototyping? | | | |-- YES --> Disable security (xpack.security.enabled=false) -- DEV ONLY | | | |-- NO --> Keep security on, set ELASTIC_PASSWORD | |-- NO (production) --> | |-- Data volume < 100 GB/day? | | |-- YES --> Single-node ES is sufficient, enable security + TLS | | |-- NO --> Multi-node cluster (3+ ES nodes) | | |-- Need high availability? | | | |-- YES --> 3 master-eligible + 2 data nodes minimum | | | |-- NO --> 3 combined master+data nodes | |-- Need TLS between all components? | |-- YES --> Use setup container with elasticsearch-certutil (see Step 2) | |-- NO --> Basic authentication only (passwords, no TLS) ``` ## Step-by-Step Guide ### 1. Set host system requirements Elasticsearch requires elevated mmap limits. This MUST be done on the Docker host, not inside the container. [src2] ```bash # Linux: set vm.max_map_count (required for Elasticsearch) sudo sysctl -w vm.max_map_count=262144 # Make persistent across reboots echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf # Windows (WSL2): run in PowerShell as admin wsl -d docker-desktop sh -c "sysctl -w vm.max_map_count=262144" # macOS (Docker Desktop): already handled by Docker Desktop VM ``` **Verify**: `sysctl vm.max_map_count` --> expected: `vm.max_map_count = 262144` ### 2. Create project directory and environment file Set up the directory structure with configuration files for each service. [src4] ```bash # Create project structure mkdir -p docker-elk/{elasticsearch,logstash/pipeline,logstash/config,kibana/config} # Create .env file with stack version and passwords cat > docker-elk/.env << 'EOF' ELASTIC_VERSION=8.17.0 ELASTIC_PASSWORD=changeme KIBANA_SYSTEM_PASSWORD=changeme LOGSTASH_INTERNAL_PASSWORD=changeme ES_MEM_LIMIT=1073741824 KB_MEM_LIMIT=1073741824 LS_MEM_LIMIT=1073741824 CLUSTER_NAME=docker-elk LICENSE=basic EOF ``` **Verify**: `cat docker-elk/.env` --> all variables should be set ### 3. Create the docker-compose.yml This is the core configuration file that defines all ELK services. [src1] Full script: [docker-compose.yml](scripts/docker-compose.yml) (89 lines) ```yaml # docker-compose.yml -- ELK Stack 8.x # Pin to identical version across all services services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION} container_name: elasticsearch # ... (see full script) ``` **Verify**: `docker compose config` --> validates the Compose file without starting services ### 4. Create Logstash pipeline configuration The Logstash pipeline defines input sources, processing filters, and the Elasticsearch output. [src3] Full script: [logstash.conf](scripts/logstash.conf) (44 lines) ```conf # logstash/pipeline/logstash.conf input { beats { port => 5044 } tcp { port => 5000 codec => json_lines } } # ... (see full script) ``` Create the Logstash settings file: ```yaml # logstash/config/logstash.yml http.host: "0.0.0.0" xpack.monitoring.elasticsearch.hosts: ["http://elasticsearch:9200"] xpack.monitoring.elasticsearch.username: "logstash_internal" xpack.monitoring.elasticsearch.password: "${LOGSTASH_INTERNAL_PASSWORD}" ``` **Verify**: `docker compose exec logstash logstash --config.test_and_exit` --> validates pipeline syntax ### 5. Create Kibana configuration Configure Kibana to connect to Elasticsearch with authentication. [src5] ```yaml # kibana/config/kibana.yml server.name: kibana server.host: "0.0.0.0" elasticsearch.hosts: ["http://elasticsearch:9200"] elasticsearch.username: "kibana_system" elasticsearch.password: "${KIBANA_SYSTEM_PASSWORD}" monitoring.ui.container.elasticsearch.enabled: true ``` **Verify**: After startup, `curl -s http://localhost:5601/api/status | jq .status.overall.level` --> `"available"` ### 6. Start the stack and initialize users Bootstrap the stack by starting Elasticsearch first, then initializing service account passwords. [src4] ```bash # Start all services cd docker-elk docker compose up -d # Wait for Elasticsearch to be healthy (30-60 seconds) until curl -s -u elastic:changeme http://localhost:9200/_cluster/health | grep -q '"status"'; do echo "Waiting for Elasticsearch..." sleep 5 done # Set kibana_system password curl -s -X POST -u elastic:changeme \ http://localhost:9200/_security/user/kibana_system/_password \ -H "Content-Type: application/json" \ -d '{"password":"changeme"}' # Create logstash_internal user with write permissions curl -s -X POST -u elastic:changeme \ http://localhost:9200/_security/user/logstash_internal \ -H "Content-Type: application/json" \ -d '{ "password":"changeme", "roles":["logstash_writer"], "full_name":"Internal Logstash User" }' # Create the logstash_writer role curl -s -X POST -u elastic:changeme \ http://localhost:9200/_security/role/logstash_writer \ -H "Content-Type: application/json" \ -d '{ "cluster":["manage_index_templates","monitor","manage_ilm"], "indices":[{ "names":["logstash-*","logs-*","metrics-*"], "privileges":["write","create_index","manage","auto_configure"] }] }' ``` **Verify**: `docker compose ps` --> all containers should show "healthy" or "running" ## Code Examples ### Complete docker-compose.yml: Single-Node Development Full script: [docker-compose.yml](scripts/docker-compose.yml) (89 lines) ```yaml # docker-compose.yml -- ELK Stack 8.x single-node development # Usage: docker compose up -d # Prereq: vm.max_map_count=262144 on host services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION} container_name: elasticsearch environment: - discovery.type=single-node - ELASTIC_PASSWORD=${ELASTIC_PASSWORD} - xpack.security.enabled=true - "ES_JAVA_OPTS=-Xms512m -Xmx512m" volumes: - es-data:/usr/share/elasticsearch/data ``` ### Logstash Pipeline: JSON + Syslog + Beats Input Full script: [logstash.conf](scripts/logstash.conf) (44 lines) ```conf # logstash/pipeline/logstash.conf # Accepts Beats (5044), TCP JSON (5000), Syslog (5140) input { beats { port => 5044 } tcp { port => 5000; codec => json_lines } syslog { port => 5140 } } ``` ### Elasticsearch Index Template: Log Data > Full script: [elasticsearch-index-template-log-data.sh](scripts/elasticsearch-index-template-log-data.sh) (26 lines) ```bash # Create an index template for structured log data # Run after Elasticsearch is healthy curl -s -X PUT -u elastic:changeme \ http://localhost:9200/_index_template/logs-template \ -H "Content-Type: application/json" \ # ... (see full script) ``` ### ILM Policy: Log Retention > Full script: [ilm-policy-log-retention.sh](scripts/ilm-policy-log-retention.sh) (31 lines) ```bash # Index Lifecycle Management policy # Hot: 30 days or 50GB --> Warm: 60 days --> Delete curl -s -X PUT -u elastic:changeme \ http://localhost:9200/_ilm/policy/logs-policy \ -H "Content-Type: application/json" \ # ... (see full script) ``` ## Anti-Patterns ### Wrong: Running without persistent volumes ```yaml # BAD -- data lost when container restarts services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:8.17.0 # No volumes defined -- all indices lost on restart ``` ### Correct: Named volumes for data persistence ```yaml # GOOD -- data survives container restarts and upgrades services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:8.17.0 volumes: - es-data:/usr/share/elasticsearch/data volumes: es-data: driver: local ``` ### Wrong: Using logstash_system for pipeline output ```yaml # BAD -- logstash_system is a monitoring-only user, cannot write indices output { elasticsearch { hosts => ["http://elasticsearch:9200"] user => "logstash_system" password => "${LOGSTASH_PASSWORD}" } } ``` ### Correct: Dedicated logstash_internal user with write role ```yaml # GOOD -- logstash_internal has the logstash_writer role with index write permissions output { elasticsearch { hosts => ["http://elasticsearch:9200"] user => "logstash_internal" password => "${LOGSTASH_INTERNAL_PASSWORD}" index => "logstash-%{+YYYY.MM.dd}" } } ``` ### Wrong: Mismatched image versions ```yaml # BAD -- mixing 8.17.0 and 8.15.0 causes version incompatibility services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:8.17.0 logstash: image: docker.elastic.co/logstash/logstash:8.15.0 kibana: image: docker.elastic.co/kibana/kibana:8.16.0 ``` ### Correct: Single version variable for all images ```yaml # GOOD -- all services pinned to the same version via .env variable services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION} logstash: image: docker.elastic.co/logstash/logstash:${ELASTIC_VERSION} kibana: image: docker.elastic.co/kibana/kibana:${ELASTIC_VERSION} # .env: ELASTIC_VERSION=8.17.0 ``` ### Wrong: Unequal JVM heap sizes ```yaml # BAD -- Xms != Xmx causes heap resizing overhead and GC pauses environment: - "ES_JAVA_OPTS=-Xms256m -Xmx2g" ``` ### Correct: Equal min and max heap ```yaml # GOOD -- equal Xms/Xmx eliminates heap resizing, 50% of container memory environment: - "ES_JAVA_OPTS=-Xms1g -Xmx1g" ``` ## Common Pitfalls - **vm.max_map_count not set**: Elasticsearch crashes on startup with `max virtual memory areas vm.max_map_count [65530] is too low`. Fix: `sudo sysctl -w vm.max_map_count=262144` on the Docker host (not inside the container). [src2] - **Elasticsearch OOM with default memory**: Docker's default memory limit is too low for Elasticsearch. Fix: Set `mem_limit` in docker-compose.yml to at least 1 GB, and `ES_JAVA_OPTS` to 50% of that (`-Xms512m -Xmx512m`). [src2] - **Kibana can't connect after security enabled**: Kibana fails with authentication errors because `kibana_system` password was not set. Fix: Use the `_security/user/kibana_system/_password` API to set the password after ES starts. [src4] - **Logstash pipeline fails silently**: Logstash starts but no data reaches Elasticsearch because the output user lacks permissions. Fix: Create a `logstash_writer` role with `write`, `create_index`, and `manage` privileges, then assign it to the `logstash_internal` user. [src3] - **Port 9200 exposed to the internet**: Docker publishes ports to 0.0.0.0 by default, exposing Elasticsearch to all network interfaces. Fix: Bind to localhost only with `127.0.0.1:9200:9200` in docker-compose.yml. [src1] - **Data directory permission errors**: Elasticsearch fails to start with permission denied on `/usr/share/elasticsearch/data`. Fix: Ensure the volume directory is owned by UID 1000 (`chown -R 1000:1000 ./es-data`). [src2] - **Logstash config not reloading**: Changes to pipeline config require Logstash restart. Fix: Enable auto-reload with `config.reload.automatic: true` in `logstash.yml` (default interval: 3 seconds). [src3] - **Kibana "Kibana server is not ready yet"**: Kibana takes 1-3 minutes to initialize after Elasticsearch is healthy. Fix: Add `depends_on` with a health check on Elasticsearch, or use a `healthcheck` on Kibana itself and wait. [src6] ## Diagnostic Commands ```bash # Check Elasticsearch cluster health curl -s -u elastic:changeme http://localhost:9200/_cluster/health?pretty # Check Elasticsearch node stats (memory, disk, CPU) curl -s -u elastic:changeme http://localhost:9200/_nodes/stats?pretty | jq '.nodes[].os' # Check Logstash pipeline status curl -s http://localhost:9600/_node/stats/pipelines?pretty # Verify Logstash can reach Elasticsearch docker compose exec logstash curl -s -u logstash_internal:changeme http://elasticsearch:9200 # Check Kibana status curl -s http://localhost:5601/api/status | jq '.status.overall' # View Elasticsearch container logs docker compose logs elasticsearch --tail=50 # View Logstash pipeline errors docker compose logs logstash --tail=50 | grep -i error # Check vm.max_map_count on the host sysctl vm.max_map_count # List all Elasticsearch indices curl -s -u elastic:changeme http://localhost:9200/_cat/indices?v # Test Logstash TCP input echo '{"message":"test log","level":"info"}' | nc localhost 5000 # Check Elasticsearch disk usage curl -s -u elastic:changeme http://localhost:9200/_cat/allocation?v # Verify TLS certificate (if enabled) curl -s --cacert ./certs/ca/ca.crt https://localhost:9200 -u elastic:changeme ``` ## Version History & Compatibility | Version | Status | Breaking Changes | Migration Notes | |---|---|---|---| | 8.17.x | Current (Jan 2026) | None | Recommended for new deployments | | 8.15.x | Supported | None | Standard upgrade path | | 8.0.x | Supported (baseline) | Security enabled by default, TLS auto-configured, Java 17 required | Requires password bootstrap on first start | | 7.17.x | Maintenance | N/A (last 7.x) | Upgrade to 8.x: enable security, update docker-compose env vars, re-index if needed | | 7.x --> 8.x | Migration | xpack.security.enabled=true by default, enrollment tokens, Kibana needs kibana_system user | Run upgrade assistant in Kibana 7.17 first, then upgrade images | ## When to Use / When Not to Use | Use When | Don't Use When | Use Instead | |---|---|---| | Centralized log aggregation for <50 GB/day | Ingesting >1 TB/day requiring dedicated hardware | Bare-metal Elastic cluster with Ansible/Terraform | | Development and testing of log pipelines | Need a managed service with SLA | Elastic Cloud, AWS OpenSearch Service | | Self-hosted observability on a single server | Only need metrics (no logs) | Prometheus + Grafana stack | | Air-gapped or on-premises deployment | Need real-time streaming analytics | Apache Kafka + Flink + Elasticsearch | | Prototyping dashboards before production | Need APM and distributed tracing only | Jaeger or Zipkin with Elasticsearch backend | ## Important Caveats - Elasticsearch 8.x generates self-signed TLS certificates on first start -- for production, replace with proper CA-signed certificates or use the `elasticsearch-certutil` tool in a setup container - The `elastic` superuser should NOT be used in production applications -- create dedicated users with minimum required privileges - Docker Compose restarts may cause Elasticsearch cluster UUID conflicts if data volumes are corrupted -- always use `docker compose down` (not `kill`) for clean shutdown - Logstash JVM heap is separate from Elasticsearch heap -- monitor both independently; Logstash typically needs 256 MB-1 GB depending on pipeline complexity - On Docker Desktop for Mac/Windows, Elasticsearch performance is significantly lower than native Linux due to the VM layer -- expect 30-50% throughput reduction - Index Lifecycle Management (ILM) policies should be configured after initial setup to prevent unbounded disk growth -- the default is no automatic cleanup ## Related Units - [Docker Compose Networking Guide](/software/devops/docker-compose-networking/2026) - [Docker Volumes & Data Persistence](/software/devops/docker-volumes-persistence/2026) - [Docker OpenSearch Stack](/software/devops/docker-opensearch-stack/2026) - [Docker Prometheus + Grafana Stack](/software/devops/docker-prometheus-grafana/2026)