--- # === IDENTITY === id: software/debugging/terraform-state-conflicts/2026 canonical_question: "How do I resolve Terraform state conflicts and lock issues?" aliases: - "Terraform state lock error" - "Error acquiring the state lock" - "Terraform force-unlock" - "Terraform state serial mismatch" - "Terraform ConditionalCheckFailedException" - "Terraform state file corruption" - "Terraform lock ID not found" - "Terraform state push conflict" - "Terraform MD5 hash mismatch" - "Terraform DynamoDB lock stuck" entity_type: software_reference domain: software > debugging > terraform_state_conflicts region: global jurisdiction: global temporal_scope: 2020-2026 # === VERIFICATION === last_verified: 2026-02-23 confidence: 0.93 version: 1.0 first_published: 2026-02-23 # === TEMPORAL VALIDITY === temporal_validity: status: stable last_breaking_change: "Terraform 1.10 (2024) — native S3 locking via use_lockfile, DynamoDB locking deprecated" next_review: 2026-08-22 change_sensitivity: medium # === CONSTRAINTS === constraints: - "Never run terraform force-unlock while another terraform process is actively running — this causes state corruption from concurrent writes" - "Always verify no other operations are in progress (check CI/CD pipelines, teammates, scheduled jobs) before force-unlocking" - "terraform state push requires local serial > remote serial — pushing an older serial overwrites newer state and loses changes" - "Local state files (.tfstate on disk) cannot be unlocked by terraform force-unlock — only remote backends support this command" - "DynamoDB-based locking is deprecated as of Terraform 1.10 — migrate to native S3 locking (use_lockfile = true) before it is removed" # === SKIP CONDITIONS === skip_this_unit_if: - condition: "Terraform plan/apply fails with provider or resource errors (not state/lock errors)" use_instead: "Terraform provider debugging or resource-specific troubleshooting" - condition: "Terraform state drift — resources exist but differ from state (no lock/conflict error)" use_instead: "terraform plan -refresh-only to reconcile drift" - condition: "Terraform init fails with backend configuration errors" use_instead: "Terraform backend configuration troubleshooting" # === AGENT HINTS === inputs_needed: - key: error_message question: "What is the exact error message? (Copy the first 2 lines of the error output)" type: choice options: ["Error acquiring the state lock", "ConditionalCheckFailedException", "Error uploading state: Conflict", "Failed to unlock state", "state serial mismatch", "Other / not sure"] - key: backend_type question: "Which Terraform backend are you using?" type: choice options: ["S3 + DynamoDB", "S3 native locking", "Terraform Cloud / HCP", "Azure Blob Storage", "GCS", "Local", "Other"] - key: context question: "What was happening when the error occurred?" type: choice options: ["CI/CD pipeline timed out or was cancelled", "Manual terraform apply was interrupted", "Two people ran terraform simultaneously", "terraform init during backend migration", "Not sure"] # === DISTRIBUTION === canonical_source: "https://knowledgelib.io/software/debugging/terraform-state-conflicts/2026" suggested_citation: "Source: knowledgelib.io — AI Knowledge Library (verified 2026-02-23)" # === RELATED UNITS === related_kos: related_to: - id: "software/debugging/docker-container-wont-start/2026" label: "Docker Container Won't Start" - id: "software/debugging/kubernetes-pod-pending/2026" label: "Kubernetes Pod Pending" solves: - id: "software/debugging/terraform-provider-errors/2026" label: "Terraform Provider Errors" # === SOURCES (7 authoritative sources) === # Types: official_docs, technical_blog, rfc_spec, academic_paper, community_resource, industry_report # Reliability: high, moderate_high, moderate, moderate_low, low, authoritative sources: - id: src1 title: "State: Locking" author: HashiCorp url: https://developer.hashicorp.com/terraform/language/state/locking type: official_docs published: 2024-11-15 reliability: authoritative - id: src2 title: "terraform force-unlock command reference" author: HashiCorp url: https://developer.hashicorp.com/terraform/cli/commands/force-unlock type: official_docs published: 2024-11-15 reliability: authoritative - id: src3 title: "Backend Type: s3" author: HashiCorp url: https://developer.hashicorp.com/terraform/language/backend/s3 type: official_docs published: 2024-11-15 reliability: authoritative - id: src4 title: "Managing Terraform State - Best Practices & Examples" author: Spacelift url: https://spacelift.io/blog/terraform-state type: technical_blog published: 2025-06-10 reliability: moderate_high - id: src5 title: "How to manage Terraform state" author: Gruntwork url: https://gruntwork.io/blog/how-to-manage-terraform-state type: technical_blog published: 2024-08-20 reliability: high - id: src6 title: "Terraform State Lock Errors: Emergency Solutions & Prevention Guide" author: Scalr url: https://scalr.com/learning-center/terraform-state-lock-errors-emergency-solutions-prevention-guide/ type: technical_blog published: 2025-03-15 reliability: moderate_high - id: src7 title: "Error uploading state: Conflict — MD5 hash mismatch" author: HashiCorp Support url: https://support.hashicorp.com/hc/en-us/articles/31856819100179 type: official_docs published: 2025-01-10 reliability: high --- # Resolve Terraform State Conflicts and Lock Issues ## TL;DR - **Bottom line**: Most Terraform state lock errors are stale locks from interrupted processes — identify the lock holder, confirm no operation is running, then `terraform force-unlock `. - **Key tool/command**: `terraform force-unlock ` - **Watch out for**: Force-unlocking while another process is actually running — this causes state corruption and potential infrastructure drift. - **Works with**: Terraform >= 0.12 (all versions), OpenTofu >= 1.6. Backend-specific: S3+DynamoDB, S3 native locking (>= 1.10), Terraform Cloud, Azure Blob, GCS. ## Constraints - Never run `terraform force-unlock` while another terraform process is actively running — this causes state corruption from concurrent writes - Always verify no other operations are in progress (check CI/CD pipelines, teammates, scheduled jobs) before force-unlocking - `terraform state push` requires local serial > remote serial — pushing an older serial overwrites newer state and loses changes - Local state files (.tfstate on disk) cannot be unlocked by `terraform force-unlock` — only remote backends support this command - DynamoDB-based locking is deprecated as of Terraform 1.10 — migrate to native S3 locking (`use_lockfile = true`) before it is removed ## Quick Reference | # | Cause | Likelihood | Signature | Fix | |---|---|---|---|---| | 1 | Stale lock from crashed/interrupted process | ~40% of cases | `Error acquiring the state lock` + lock info with old timestamp | `terraform force-unlock ` | | 2 | CI/CD pipeline killed mid-apply | ~20% of cases | Lock info shows CI runner hostname, pipeline was cancelled/timed out | Kill orphan process, then `terraform force-unlock ` | | 3 | Concurrent terraform runs (no CI mutex) | ~12% of cases | Two pipelines or users running simultaneously | Wait for first to complete, add CI concurrency controls | | 4 | DynamoDB lock table permission denied | ~8% of cases | `ConditionalCheckFailedException` or AccessDeniedException | Fix IAM: add `dynamodb:PutItem`, `dynamodb:DeleteItem` on lock table | | 5 | State serial mismatch after state push | ~6% of cases | `Error uploading state: Conflict` + MD5 hash mismatch | `terraform state pull > tmp.tfstate`, increment serial, `terraform state push tmp.tfstate` | | 6 | Backend migration lock conflict | ~5% of cases | `Error acquiring the state lock` during `terraform init -migrate-state` | Complete or abort previous migration, force-unlock, retry init | | 7 | S3 lock file not cleaned up | ~3% of cases | `.tflock` file persists in S3 bucket after crash | Delete `*.tflock` file from S3 bucket manually | | 8 | Network interruption during apply | ~2% of cases | Process exited uncleanly, lock remains in remote backend | Verify process is dead, then `terraform force-unlock ` | | 9 | Terraform Cloud workspace locked | ~2% of cases | Workspace shows "Locked" in UI, API returns 409 | Unlock via UI or `POST /api/v2/workspaces/{id}/actions/unlock` | | 10 | Azure Blob lease not released | ~1% of cases | `Error: Error locking state: Error acquiring the state lock` with Azure lease ID | Break blob lease via Azure CLI: `az storage blob lease break` | | 11 | Lock ID mismatch on unlock attempt | ~1% of cases | `Failed to unlock state: lock ID does not match existing lock ID` | Get correct lock ID from error output or backend, retry | ## Decision Tree ``` START ├── Error message contains "Error acquiring the state lock"? │ ├── YES → Is another terraform process running (check CI/CD, teammates)? │ │ ├── YES → Wait for it to finish, or add -lock-timeout=10m │ │ └── NO → Extract LOCK_ID from error → terraform force-unlock │ └── NO ↓ ├── Error message contains "ConditionalCheckFailedException"? │ ├── YES → Check DynamoDB IAM permissions (PutItem, DeleteItem, GetItem) │ │ ├── Permissions OK → Stale DynamoDB lock → delete item from lock table │ │ └── Permissions missing → Fix IAM policy │ └── NO ↓ ├── Error message contains "Error uploading state: Conflict" or "MD5 hash"? │ ├── YES → Serial mismatch → pull state, increment serial by 2, push back │ └── NO ↓ ├── Error message contains "Failed to unlock state: lock ID does not match"? │ ├── YES → Extract correct lock ID from backend → retry force-unlock │ └── NO ↓ ├── Using Terraform Cloud and workspace shows "Locked"? │ ├── YES → Unlock via UI or API POST /actions/unlock │ └── NO ↓ └── DEFAULT → Run terraform init -reconfigure, then retry operation ``` ## Step-by-Step Guide ### 1. Identify the lock holder and error type Read the full error message. Terraform prints the lock ID, who holds it, when it was acquired, and what operation they were running. [src1] ```bash # The error output looks like this: # Error: Error acquiring the state lock # # Error message: ConditionalCheckFailedException: The conditional request failed # Lock Info: # ID: a1b2c3d4-e5f6-7890-abcd-ef1234567890 # Path: terraform-state/production/terraform.tfstate # Operation: OperationTypeApply # Who: runner@ci-node-42 # Version: 1.9.5 # Created: 2026-02-23 10:15:32.123456 +0000 UTC # Info: ``` **Verify**: Check that the `Who` field and `Created` timestamp — if the lock is hours/days old, it is stale. ### 2. Confirm no active operations Before force-unlocking, ensure no other process is running terraform against this state file. [src6] ```bash # Check local processes ps aux | grep terraform # Check CI/CD pipelines (GitHub Actions example) gh run list --workflow=terraform.yml --status=in_progress # Check Terraform Cloud (if applicable) curl -s -H "Authorization: Bearer $TFC_TOKEN" \ "https://app.terraform.io/api/v2/workspaces/$WORKSPACE_ID/runs?filter[status]=applying" \ | jq '.data[] | {id: .id, status: .attributes.status}' ``` **Verify**: No terraform processes listed, no CI runs in progress. ### 3. Force-unlock the state Use the lock ID from the error message. [src2] ```bash # Standard force-unlock (will prompt for confirmation) terraform force-unlock a1b2c3d4-e5f6-7890-abcd-ef1234567890 # Skip confirmation prompt (CI/CD usage) terraform force-unlock -force a1b2c3d4-e5f6-7890-abcd-ef1234567890 ``` **Verify**: `terraform plan` → runs without lock errors. ### 4. Handle serial mismatch conflicts If you get an MD5 hash or serial conflict when pushing state, increment the serial. [src7] ```bash # Pull current remote state terraform state pull > current-state.tfstate # Check current serial number grep '"serial"' current-state.tfstate # Output: "serial": 42, # Edit the serial to be at least 2 higher than remote # (use jq or manual edit) jq '.serial = .serial + 2' current-state.tfstate > fixed-state.tfstate # Push the fixed state terraform state push fixed-state.tfstate # Clean up temporary files rm current-state.tfstate fixed-state.tfstate ``` **Verify**: `terraform plan` → shows expected changes (or no changes). ### 5. Delete stale DynamoDB lock (S3 backend last resort) If force-unlock fails, delete the lock item directly from DynamoDB. [src3] ```bash # Scan for stale locks aws dynamodb scan \ --table-name terraform-state-locks \ --filter-expression "attribute_exists(LockID)" \ --projection-expression "LockID,Info" # Delete the stale lock entry aws dynamodb delete-item \ --table-name terraform-state-locks \ --key '{"LockID": {"S": "terraform-state/production/terraform.tfstate-md5"}}' ``` **Verify**: `terraform plan` → lock acquired successfully. ### 6. Recover from state corruption If state is corrupted (empty, truncated, or invalid JSON), restore from S3 versioning. [src5] ```bash # List state file versions in S3 aws s3api list-object-versions \ --bucket my-terraform-state \ --prefix "production/terraform.tfstate" \ --query 'Versions[0:5].{VersionId:VersionId,LastModified:LastModified,Size:Size}' # Download the last known good version aws s3api get-object \ --bucket my-terraform-state \ --key "production/terraform.tfstate" \ --version-id "abc123def456" \ restored-state.tfstate # Verify it is valid JSON python3 -c "import json; json.load(open('restored-state.tfstate')); print('Valid')" # Push restored state (increment serial first) jq '.serial = .serial + 2' restored-state.tfstate > push-state.tfstate terraform state push push-state.tfstate ``` **Verify**: `terraform plan` → shows expected infrastructure state, no unknown resources. ## Code Examples ### HCL: S3 backend with native locking (recommended) ```hcl # Input: Backend config in terraform block # Output: Remote state with S3-native locking (no DynamoDB needed) terraform { backend "s3" { bucket = "mycompany-terraform-state" key = "production/network/terraform.tfstate" region = "us-east-1" encrypt = true use_lockfile = true # Native S3 locking (Terraform >= 1.10) } } ``` ### HCL: S3 backend with DynamoDB locking (legacy) ```hcl # Input: Backend config with DynamoDB lock table # Output: Remote state with DynamoDB-based locking (deprecated) terraform { backend "s3" { bucket = "mycompany-terraform-state" key = "production/network/terraform.tfstate" region = "us-east-1" encrypt = true dynamodb_table = "terraform-state-locks" # Deprecated in 1.10 } } ``` ### Bash: CI/CD safe apply with lock timeout and retry > Full script: [bash-ci-cd-safe-apply-with-lock-timeout-and-retry.sh](scripts/bash-ci-cd-safe-apply-with-lock-timeout-and-retry.sh) (29 lines) ```bash #!/usr/bin/env bash # Input: Terraform workspace directory # Output: Safe apply with lock timeout, retry, and auto-unlock on stale lock set -euo pipefail MAX_RETRIES=3 # ... (see full script) ``` ### Bash: GitHub Actions concurrency control ```yaml # Input: GitHub Actions workflow file # Output: Single-concurrent terraform apply per environment name: Terraform Apply on: push: branches: [main] concurrency: group: terraform-${{ github.ref }}-production cancel-in-progress: false # Never cancel running apply jobs: apply: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: hashicorp/setup-terraform@v3 - run: terraform init - run: terraform apply -auto-approve -lock-timeout=10m ``` ## Anti-Patterns ### Wrong: Force-unlocking without checking for active operations ```bash # BAD — blindly force-unlocking can corrupt state if another process is running terraform force-unlock -force $(terraform plan 2>&1 | grep -oP 'ID:\s+\K[a-f0-9-]+') ``` ### Correct: Verify no active operations, then unlock ```bash # GOOD — check for running processes first ps aux | grep "[t]erraform" # Check local gh run list --workflow=terraform.yml --status=in_progress # Check CI # Only after confirming nothing is running: terraform force-unlock a1b2c3d4-e5f6-7890-abcd-ef1234567890 ``` ### Wrong: Disabling state locking to avoid lock errors ```hcl # BAD — disabling locking creates race conditions and state corruption terraform { backend "s3" { bucket = "my-state" key = "terraform.tfstate" region = "us-east-1" # No lock table, no use_lockfile — concurrent applies WILL corrupt state } } ``` ### Correct: Enable locking and handle contention with timeouts ```hcl # GOOD — always enable locking, use timeouts for contention terraform { backend "s3" { bucket = "my-state" key = "terraform.tfstate" region = "us-east-1" encrypt = true use_lockfile = true # Native S3 locking } } # Run with: terraform apply -lock-timeout=10m ``` ### Wrong: Using terraform state push without checking serial ```bash # BAD — pushing without checking serial can overwrite newer state terraform state pull > backup.tfstate # ... make manual edits ... terraform state push backup.tfstate # Error: serial 40 is not greater than current serial 42 ``` ### Correct: Pull latest, increment serial, then push ```bash # GOOD — always pull fresh, increment serial, verify before pushing terraform state pull > current.tfstate SERIAL=$(jq '.serial' current.tfstate) jq ".serial = $SERIAL + 2" current.tfstate > updated.tfstate terraform state push updated.tfstate ``` ### Wrong: Storing state in Git for "version control" ```bash # BAD — state contains secrets and Git has no locking git add terraform.tfstate git commit -m "Update state" # Merge conflicts on .tfstate are unrecoverable ``` ### Correct: Use remote backend with encryption and versioning ```hcl # GOOD — remote backend with encryption, versioning, and locking terraform { backend "s3" { bucket = "my-state" key = "terraform.tfstate" region = "us-east-1" encrypt = true use_lockfile = true } } # S3 versioning provides the version history; Git tracks only .tf files ``` ## Common Pitfalls - **Forgetting -lock-timeout in CI/CD**: Default lock timeout is 0s — if another pipeline holds the lock, terraform immediately fails. Fix: Always add `-lock-timeout=5m` to plan/apply in CI/CD. [src6] - **DynamoDB table with wrong key name**: The partition key must be named exactly `LockID` (capital L, capital ID) with type String. Any other name silently fails to lock. Fix: Verify with `aws dynamodb describe-table --table-name terraform-state-locks`. [src3] - **S3 bucket without versioning**: If state gets corrupted and there is no versioning, the only recovery path is `terraform import` for every resource. Fix: Enable versioning at bucket creation: `aws s3api put-bucket-versioning --bucket my-state --versioning-configuration Status=Enabled`. [src5] - **Multiple backends referencing same state file**: Two Terraform configs pointing to the same S3 key create implicit coupling and lock contention. Fix: Use unique keys per stack/environment: `key = "env/${var.env}/component/terraform.tfstate"`. [src4] - **Running terraform commands from different directories**: Each directory is a separate Terraform root module with its own state. Running `terraform plan` from the wrong directory operates on the wrong state. Fix: Always verify with `terraform state list` before modifying. [src1] - **Terraform Cloud workspace auto-lock on VCS-driven runs**: VCS-triggered runs lock the workspace. Manual CLI runs fail with lock errors. Fix: Queue destroy/unlock via UI, or use API to force-unlock. [src7] - **Ignoring lock info timestamps**: A lock from 5 minutes ago might be a running apply. A lock from 5 hours ago is almost certainly stale. Fix: Always check the `Created` field in the lock info before force-unlocking. [src2] ## Diagnostic Commands > Full script: [diagnostic-commands.sh](scripts/diagnostic-commands.sh) (28 lines) ```bash # Check who holds the state lock (error output includes lock info) terraform plan 2>&1 | grep -A 10 "Lock Info" # Inspect current state serial and lineage terraform state pull | jq '{serial, lineage, terraform_version}' # List all resources in current state # ... (see full script) ``` ## Version History & Compatibility | Version | Status | Breaking Changes | Migration Notes | |---|---|---|---| | Terraform >= 1.10 | Current | Native S3 locking (`use_lockfile`), DynamoDB locking deprecated | Add `use_lockfile = true`, keep `dynamodb_table` for migration period | | Terraform 1.0-1.9 | Supported | None for locking | S3 backend requires `dynamodb_table` for locking | | Terraform 0.14-0.15 | Legacy | Backend config changes | `terraform init -reconfigure` required after upgrades | | Terraform 0.12-0.13 | EOL | HCL2 migration | State format compatible, backends unchanged | | OpenTofu >= 1.6 | Current (fork) | Fork of Terraform, same state format | `force-unlock`, state commands work identically | ## When to Use / When Not to Use | Use When | Don't Use When | Use Instead | |---|---|---| | `Error acquiring the state lock` appears | State drift without lock errors | `terraform plan -refresh-only` | | CI/CD pipeline stuck waiting for lock | Provider authentication failures | Provider-specific documentation | | `terraform force-unlock` needed | Terraform plan shows unexpected changes | Manual state inspection / `terraform import` | | State serial/MD5 mismatch errors | Terraform version upgrade issues | Terraform upgrade guide | | Recovering from corrupted state file | Infrastructure is wrong but state is correct | `terraform apply` to converge | ## Important Caveats - Native S3 locking (`use_lockfile`) stores a `.tflock` file alongside the state — IAM policies must grant `s3:PutObject` and `s3:DeleteObject` on `*.tflock` paths, not just the state file itself - `terraform force-unlock` behavior varies by backend — some backends (like Azure Blob Storage) require breaking the lease separately from the Terraform lock - State files contain sensitive data (passwords, tokens, private keys from provider outputs) in plaintext — always encrypt at rest (S3 SSE, Azure Storage encryption) and restrict access via IAM - After any manual state manipulation (`state push`, `state mv`, `state rm`), always run `terraform plan` to verify the state matches expected infrastructure before running `apply` - Terraform Cloud/Enterprise serial conflicts often require API-level state manipulation — the CLI `state push` may fail due to client-side validation differences between local and remote Terraform versions ## Related Units - [Docker Container Won't Start](/software/debugging/docker-container-wont-start/2026) - [Kubernetes Pod Pending](/software/debugging/kubernetes-pod-pending/2026)