--- # === IDENTITY === id: software/security/secrets-management/2026 canonical_question: "What are the best practices for secrets management?" aliases: - "secrets management best practices" - "how to store API keys securely" - "HashiCorp Vault secrets management" - "AWS Secrets Manager vs Vault" - "prevent hardcoded credentials" - "CWE-798 prevention" - "secure credential storage" - "secrets rotation automation" entity_type: software_reference domain: software > security > Secrets Management region: global jurisdiction: global temporal_scope: 2024-2026 # === VERIFICATION === last_verified: 2026-02-27 confidence: 0.94 version: 1.0 first_published: 2026-02-27 # === TEMPORAL VALIDITY === temporal_validity: status: stable last_breaking_change: "HashiCorp Vault 1.20 GA (June 2025); AWS Secrets Manager cross-account sharing (2024); SOPS donated to CNCF Sandbox (2023)" next_review: 2026-08-26 change_sensitivity: medium # === CONSTRAINTS === constraints: - "NEVER hardcode secrets in source code, configuration files, or container images -- CWE-798" - "NEVER commit secrets to version control -- even in private repos, git history retains them permanently" - "Secrets MUST be encrypted at rest and in transit -- use TLS for all secrets manager communication" - "Rotate secrets automatically on a schedule appropriate to sensitivity -- API keys quarterly, database credentials monthly minimum" - "Apply least-privilege access to every secret -- no blanket read-all permissions" - "NEVER log secret values -- mask or redact in all application and infrastructure logs" # === SKIP CONDITIONS === skip_this_unit_if: - condition: "Need to store user passwords (hashing, not secrets management)" use_instead: "software/security/password-hashing/2026" - condition: "Need encryption key management (KMS) rather than secret storage" use_instead: "software/security/encryption-key-management/2026" - condition: "Need OAuth/OIDC token management for end users" use_instead: "software/patterns/oauth-implementation/2026" # === AGENT HINTS === inputs_needed: - key: "infrastructure" question: "What infrastructure or cloud provider are you using?" type: choice options: ["AWS", "GCP", "Azure", "Multi-cloud", "On-premises", "Kubernetes", "Other"] - key: "secret_type" question: "What type of secrets do you need to manage?" type: choice options: ["API keys", "Database credentials", "TLS certificates", "SSH keys", "Environment variables", "Multiple types"] # === DISTRIBUTION === canonical_source: "https://knowledgelib.io/software/security/secrets-management/2026" suggested_citation: "Source: knowledgelib.io -- AI Knowledge Library (verified 2026-02-27)" # === RELATED UNITS === related_kos: depends_on: [] related_to: - id: "software/security/xss-prevention/2026" label: "XSS Prevention" - id: "software/devops/docker-security/2026" label: "Docker Security Best Practices" solves: - id: "software/security/credential-leak-response/2026" label: "Credential Leak Incident Response" alternative_to: [] often_confused_with: - id: "software/security/encryption-key-management/2026" label: "Encryption Key Management (KMS)" - id: "software/security/password-hashing/2026" label: "Password Hashing for User Authentication" # === SOURCES (7 authoritative sources) === sources: - id: src1 title: "Secrets Management Cheat Sheet" author: OWASP Foundation url: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html type: community_resource published: 2025-01-15 reliability: authoritative - id: src2 title: "Vault Documentation" author: HashiCorp url: https://developer.hashicorp.com/vault/docs type: official_docs published: 2026-01-07 reliability: authoritative - id: src3 title: "AWS Secrets Manager User Guide" author: Amazon Web Services url: https://docs.aws.amazon.com/secretsmanager/latest/userguide/ type: official_docs published: 2025-12-01 reliability: authoritative - id: src4 title: "CWE-798: Use of Hard-coded Credentials" author: MITRE url: https://cwe.mitre.org/data/definitions/798.html type: official_docs published: 2024-12-01 reliability: authoritative - id: src5 title: "The Twelve-Factor App: III. Config" author: Adam Wiggins url: https://12factor.net/config type: technical_blog published: 2017-01-30 reliability: high - id: src6 title: "git-secrets: Prevents committing secrets to git repositories" author: AWS Labs url: https://github.com/awslabs/git-secrets type: official_docs published: 2025-06-01 reliability: high - id: src7 title: "SOPS: Simple and flexible tool for managing secrets" author: CNCF / Mozilla url: https://github.com/getsops/sops type: official_docs published: 2025-10-01 reliability: high --- # Secrets Management: Best Practices and Tool Guide ## TL;DR - **Bottom line**: Centralize secrets in a dedicated secrets manager (Vault, AWS SM, or cloud-native equivalent), automate rotation, enforce least-privilege access, and use pre-commit hooks to prevent secrets from ever reaching version control. - **Key tool/command**: `vault kv get secret/myapp/db` (HashiCorp Vault) or `aws secretsmanager get-secret-value --secret-id myapp/db` (AWS Secrets Manager) - **Watch out for**: Hardcoded secrets in source code (CWE-798) -- the #1 cause of credential leaks, ranked #14 in the 2024 CWE Top 25. - **Works with**: All major clouds (AWS, GCP, Azure), Kubernetes (External Secrets Operator), CI/CD pipelines, and every programming language via SDKs. ## Constraints - NEVER hardcode secrets in source code, configuration files, or container images -- CWE-798 - NEVER commit secrets to version control -- even in private repos, git history retains them permanently - Secrets MUST be encrypted at rest and in transit -- use TLS for all secrets manager communication - Rotate secrets automatically on a schedule appropriate to sensitivity -- API keys quarterly, database credentials monthly minimum - Apply least-privilege access to every secret -- no blanket read-all permissions - NEVER log secret values -- mask or redact in all application and infrastructure logs ## Quick Reference ### Tool Comparison | Tool | Type | Best For | Rotation | Cost | Complexity | Cloud Lock-in | |---|---|---|---|---|---|---| | HashiCorp Vault | Self-hosted / HCP | Multi-cloud, enterprise, dynamic secrets | Built-in (dynamic) | Free OSS / HCP from $0.03/hr | High | None | | AWS Secrets Manager | Managed service | AWS-native workloads | Built-in (Lambda) | $0.40/secret/mo + $0.05/10K API | Low | AWS | | GCP Secret Manager | Managed service | GCP-native workloads | Custom (Cloud Functions) | $0.06/active version/mo | Low | GCP | | Azure Key Vault | Managed service | Azure-native workloads | Built-in (certificates) | $0.03/10K ops (Standard) | Low | Azure | | SOPS (CNCF) | File encryption | GitOps, encrypted config in repos | Manual | Free OSS | Medium | None | | git-crypt | File encryption | Simple repo encryption | Manual | Free OSS | Low | None | | dotenv (.env) | Local file | Local development only | Manual | Free | Low | None | ### Secret Types and Recommended Storage | Secret Type | Recommended Storage | Rotation Frequency | Notes | |---|---|---|---| | Database credentials | Vault dynamic secrets / cloud SM | Per-connection or monthly | Prefer dynamic (ephemeral) credentials | | API keys (third-party) | Cloud secrets manager | Quarterly or on compromise | Store with metadata (expiry, owner, scope) | | TLS certificates | Vault PKI / cloud certificate manager | 90 days (Let's Encrypt) or annually | Automate with cert-manager in K8s | | SSH keys | Vault SSH secrets engine | Per-session or daily | Signed certificates preferred over static keys | | Encryption keys | Cloud KMS (not secrets manager) | Annually or per policy | Use envelope encryption -- never store KEK with DEK | | CI/CD tokens | CI/CD built-in secrets + short TTL | Per-pipeline run | Avoid long-lived tokens; use OIDC federation | | .env files | Local only, never committed | N/A | Add to .gitignore; use secrets manager for production | ## Decision Tree ``` START: What is your deployment environment? ├── Single cloud (AWS/GCP/Azure)? │ ├── YES → Use cloud-native secrets manager (AWS SM, GCP SM, Azure KV) │ └── NO ↓ ├── Multi-cloud or hybrid? │ ├── YES → Use HashiCorp Vault (self-hosted or HCP) │ └── NO ↓ ├── Kubernetes-only? │ ├── YES → External Secrets Operator + cloud SM or Vault │ └── NO ↓ ├── Small team, few secrets, GitOps workflow? │ ├── YES → SOPS + cloud KMS for encryption keys │ └── NO ↓ ├── Local development only? │ ├── YES → dotenv (.env files in .gitignore) + git-secrets pre-commit hook │ └── NO ↓ └── DEFAULT → HashiCorp Vault (most flexible, no lock-in) ``` ## Step-by-Step Guide ### 1. Install pre-commit secret detection Prevent secrets from ever reaching version control. Install git-secrets or detect-secrets as a pre-commit hook. [src6] ```bash # Install git-secrets (AWS Labs) brew install git-secrets # macOS # or: git clone https://github.com/awslabs/git-secrets && cd git-secrets && make install # Initialize hooks in your repo cd /path/to/your/repo git secrets --install git secrets --register-aws # adds AWS-specific patterns # Add custom patterns (API keys, generic secrets) git secrets --add '[A-Za-z0-9]{32,}' # generic long tokens git secrets --add 'PRIVATE KEY' git secrets --add 'password\s*=\s*["\047][^"\047]+' # Test: this commit should be rejected echo "AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" >> test.txt git add test.txt && git commit -m "test" # should fail ``` **Verify**: `git secrets --scan` → should report any existing secrets in the repo. ### 2. Set up a centralized secrets manager Choose based on your infrastructure (see Decision Tree). Example: HashiCorp Vault dev server for evaluation. [src2] ```bash # Start Vault dev server (evaluation only -- NOT for production) vault server -dev -dev-root-token-id="dev-only-token" # In another terminal, set environment export VAULT_ADDR='http://127.0.0.1:8200' export VAULT_TOKEN='dev-only-token' # Enable KV secrets engine v2 vault secrets enable -path=secret kv-v2 # Store a secret vault kv put secret/myapp/db \ username="dbadmin" \ password="s3cur3-p@ssw0rd" \ host="db.example.com" \ port="5432" # Read it back vault kv get -format=json secret/myapp/db ``` **Verify**: `vault kv get secret/myapp/db` → should display the stored key-value pairs. ### 3. Configure application to fetch secrets at runtime Replace hardcoded credentials with secrets manager SDK calls. Never bake secrets into build artifacts. [src1] ```python # Python: Fetch secret from Vault at application startup import hvac # pip install hvac>=2.4.0 client = hvac.Client( url='https://vault.example.com:8200', token=os.environ.get('VAULT_TOKEN') # token from env, not hardcoded ) secret = client.secrets.kv.v2.read_secret_version( path='myapp/db', mount_point='secret' ) db_password = secret['data']['data']['password'] ``` **Verify**: Application starts successfully and connects to the database without any credentials in source code. ### 4. Implement automated secret rotation Configure automatic rotation to limit the blast radius of compromised credentials. [src3] ```bash # AWS Secrets Manager: Enable rotation with a Lambda function aws secretsmanager rotate-secret \ --secret-id myapp/db-credentials \ --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789:function:SecretsRotation \ --rotation-rules '{"AutomaticallyAfterDays": 30}' # Vault: Dynamic database credentials (auto-expire) vault write database/config/mydb \ plugin_name=postgresql-database-plugin \ connection_url="postgresql://{{username}}:{{password}}@db.example.com:5432/mydb" \ allowed_roles="myapp-role" \ username="vault_admin" \ password="vault_admin_password" vault write database/roles/myapp-role \ db_name=mydb \ creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';" \ default_ttl="1h" \ max_ttl="24h" ``` **Verify**: `vault read database/creds/myapp-role` → generates a new ephemeral credential with a 1h TTL. ### 5. Encrypt secrets in configuration files (GitOps) For GitOps workflows, encrypt secrets in-repo with SOPS so they can be safely committed. [src7] ```bash # Install SOPS brew install sops # macOS # or download from https://github.com/getsops/sops/releases # Create a .sops.yaml config (use AWS KMS, GCP KMS, or age key) cat > .sops.yaml << 'EOF' creation_rules: - path_regex: secrets/.*\.yaml$ kms: "arn:aws:kms:us-east-1:123456789:key/abcd-1234-efgh" - path_regex: secrets/.*\.env$ age: "age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p" EOF # Encrypt a secrets file (values only, keys stay readable) sops --encrypt secrets/production.yaml > secrets/production.enc.yaml # Decrypt at deploy time sops --decrypt secrets/production.enc.yaml > /tmp/production.yaml ``` **Verify**: `cat secrets/production.enc.yaml` → keys are readable, values are encrypted. `sops --decrypt` recovers original values. ## Code Examples ### Python: HashiCorp Vault with hvac ```python # Input: VAULT_ADDR and VAULT_TOKEN environment variables # Output: Database connection using dynamically fetched credentials import os import hvac # pip install hvac>=2.4.0 def get_vault_client(): """Create authenticated Vault client.""" client = hvac.Client( url=os.environ['VAULT_ADDR'], token=os.environ['VAULT_TOKEN'] ) if not client.is_authenticated(): raise RuntimeError("Vault authentication failed") return client def get_db_credentials(client, path='myapp/db'): """Fetch database credentials from Vault KV v2.""" resp = client.secrets.kv.v2.read_secret_version( path=path, mount_point='secret' ) return resp['data']['data'] # {'username': ..., 'password': ...} ``` ### Node.js: AWS Secrets Manager ```javascript // Input: AWS credentials (IAM role or env vars) + secret name // Output: Parsed secret object with database credentials const { SecretsManagerClient, GetSecretValueCommand } = require("@aws-sdk/client-secrets-manager"); // ^3.500.0 async function getSecret(secretId, region = "us-east-1") { const client = new SecretsManagerClient({ region }); const resp = await client.send( new GetSecretValueCommand({ SecretId: secretId }) ); return JSON.parse(resp.SecretString); } // Usage: const creds = await getSecret("myapp/db-credentials"); ``` ### Python: AWS Secrets Manager with boto3 ```python # Input: AWS credentials (IAM role or env vars) + secret name # Output: Parsed secret dict with credentials import json import boto3 # pip install boto3>=1.34.0 def get_secret(secret_name, region="us-east-1"): """Fetch and parse a secret from AWS Secrets Manager.""" client = boto3.client("secretsmanager", region_name=region) resp = client.get_secret_value(SecretId=secret_name) return json.loads(resp["SecretString"]) # Usage: creds = get_secret("myapp/db-credentials") # Returns: {"username": "admin", "password": "s3cur3", "host": "..."} ``` ### Bash: .env Best Practices for Local Development ```bash # Input: .env file in project root # Output: Environment variables loaded into shell session # 1. Create .env file (NEVER commit this) cat > .env << 'EOF' DB_HOST=localhost DB_USER=devuser DB_PASS=local-dev-only-password API_KEY=dev-test-key-not-real EOF # 2. Add to .gitignore IMMEDIATELY echo ".env" >> .gitignore echo ".env.*" >> .gitignore echo "!.env.example" >> .gitignore # 3. Create a .env.example with placeholder values (safe to commit) cat > .env.example << 'EOF' DB_HOST=localhost DB_USER=your_username DB_PASS=your_password API_KEY=your_api_key EOF # 4. Load in shell (bash/zsh) set -a; source .env; set +a ``` ## Anti-Patterns ### Wrong: Hardcoded secrets in source code ```python # BAD -- CWE-798: hardcoded credentials # Anyone with repo access (or a decompiler) gets your production password import psycopg2 conn = psycopg2.connect( host="prod-db.example.com", user="admin", password="SuperSecret123!" # hardcoded -- NEVER do this ) ``` ### Correct: Fetch secrets from a secrets manager ```python # GOOD -- credentials fetched at runtime from Vault import os, hvac, psycopg2 client = hvac.Client(url=os.environ['VAULT_ADDR'], token=os.environ['VAULT_TOKEN']) creds = client.secrets.kv.v2.read_secret_version(path='myapp/db')['data']['data'] conn = psycopg2.connect( host=creds['host'], user=creds['username'], password=creds['password'] ) ``` ### Wrong: Secrets committed to git history ```bash # BAD -- even if you delete the file, git history retains the secret forever echo "API_KEY=sk-live-abc123xyz" > config.env git add config.env && git commit -m "add config" # Later: git rm config.env # TOO LATE -- secret is in git log ``` ### Correct: Pre-commit hooks + .gitignore ```bash # GOOD -- prevent secrets from ever entering the repo echo "*.env" >> .gitignore echo "!*.env.example" >> .gitignore git secrets --install # install pre-commit hook git secrets --register-aws # Any commit containing AWS keys is now automatically blocked ``` ### Wrong: Secrets in Docker image layers ```dockerfile # BAD -- secret is baked into the image layer and visible via docker history FROM node:20-alpine ENV DATABASE_URL="postgresql://admin:secret@db:5432/myapp" COPY . /app RUN npm install CMD ["node", "server.js"] ``` ### Correct: Inject secrets at runtime ```dockerfile # GOOD -- no secrets in the image; inject at runtime FROM node:20-alpine COPY . /app RUN npm install # Secret injected via: docker run -e DATABASE_URL="$(vault kv get -field=url secret/myapp/db)" myapp CMD ["node", "server.js"] ``` ### Wrong: Environment variables visible in process listing ```bash # BAD -- env vars are visible to any process on the host via /proc/PID/environ docker run -e DB_PASSWORD="secret123" myapp # Any user on the host can: cat /proc//environ | tr '\0' '\n' ``` ### Correct: Mount secrets as files in tmpfs ```yaml # GOOD -- Kubernetes: mount from secrets manager via sidecar, tmpfs volume apiVersion: v1 kind: Pod spec: volumes: - name: secrets emptyDir: medium: Memory # tmpfs -- never written to disk containers: - name: app volumeMounts: - name: secrets mountPath: /mnt/secrets readOnly: true - name: vault-agent # sidecar fetches + renews secrets image: hashicorp/vault:1.20 volumeMounts: - name: secrets mountPath: /mnt/secrets ``` ## Common Pitfalls - **Forgetting .gitignore before first commit**: Once `.env` is committed, removing it from `.gitignore` does not remove it from git history. Fix: Add `.env` to `.gitignore` in the initial commit. If already committed, use `git filter-branch` or BFG Repo-Cleaner, then rotate all exposed secrets. [src4] - **Using Parameter Store instead of Secrets Manager on AWS**: AWS SSM Parameter Store lacks built-in rotation and has lower throughput limits (40 TPS vs 10,000 TPS for Secrets Manager). Fix: Use Secrets Manager for credentials that need rotation; Parameter Store for non-sensitive config. [src3] - **Single long-lived Vault token**: A root or long-lived token defeats Vault's security model. Fix: Use AppRole, Kubernetes, or OIDC auth methods that issue short-lived tokens with specific policies. [src2] - **Not masking secrets in CI/CD logs**: Build logs often contain expanded environment variables. Fix: Use your CI/CD platform's secret masking feature (GitHub Actions `::add-mask::`, GitLab CI masked variables). [src1] - **Storing encryption keys alongside encrypted data**: Defeats the purpose of encryption. Fix: Use envelope encryption -- data encrypted with DEK, DEK encrypted with KEK stored in separate KMS. [src1] - **Sharing secrets via Slack/email**: Secrets sent through messaging platforms persist in message history and backups. Fix: Use a secrets manager's one-time-share feature or a tool like Hashicorp Vault's cubbyhole response wrapping. [src1] - **No secret expiry or rotation**: Static secrets accumulate risk over time. Fix: Set TTLs on all secrets; use Vault dynamic secrets for database credentials to get per-connection ephemeral credentials. [src2] - **Overly broad IAM policies for secrets access**: Granting `secretsmanager:GetSecretValue` on `*` allows any compromised service to read all secrets. Fix: Scope IAM policies to specific secret ARNs with condition keys. [src3] ## Diagnostic Commands ```bash # Check for hardcoded secrets in a repository git secrets --scan # or use detect-secrets: detect-secrets scan --all-files # Check for high-entropy strings (potential secrets) with trufflehog trufflehog git file://. --only-verified # Verify Vault connectivity and authentication vault status vault token lookup # List all secrets in a Vault KV path vault kv list secret/myapp/ # Check AWS Secrets Manager secret metadata aws secretsmanager describe-secret --secret-id myapp/db-credentials # View secret rotation status (AWS) aws secretsmanager describe-secret --secret-id myapp/db-credentials \ --query '{RotationEnabled: RotationEnabled, LastRotated: LastRotatedDate, NextRotation: NextRotationDate}' # Scan Docker image for embedded secrets docker history --no-trunc myapp:latest | grep -i -E 'password|secret|key|token' # Check Kubernetes secrets (base64-encoded, not encrypted) kubectl get secret myapp-secret -o jsonpath='{.data}' | jq -r 'to_entries[] | "\(.key): \(.value | @base64d)"' # Verify SOPS encryption sops --decrypt secrets/production.enc.yaml > /dev/null && echo "Decryption OK" ``` ## Version History & Compatibility | Tool | Version | Status | Key Change | |---|---|---|---| | HashiCorp Vault | 1.20.x | Current (GA June 2025) | Secrets Operator for K8s, IBM RACF support | | HashiCorp Vault | 1.18.x | LTS | Vault Proxy, Secrets Sync | | AWS Secrets Manager | Current | GA | Cross-account sharing, Lambda rotation v2 | | GCP Secret Manager | v1 | GA | Regional replication, IAM conditions | | Azure Key Vault | Current | GA | RBAC model (vs access policies), soft-delete default | | SOPS | 3.9.x | Current (CNCF Sandbox) | age encryption support, audit logging | | git-secrets | 1.3.x | Maintained | pre-commit framework integration | | hvac (Python) | 2.4.x | Current | Python 3.8+, KV v2 improvements | | External Secrets Operator | 0.10.x | Current | Multi-provider, PushSecret CRD | ## When to Use / When Not to Use | Use When | Don't Use When | Use Instead | |---|---|---| | Any production credentials (DB, API keys, tokens) | Storing non-sensitive config (feature flags, app settings) | Environment variables or config files | | Multi-service architecture sharing credentials | Single-developer local prototype | .env file with .gitignore | | Compliance requires audit trail for credential access | Storing user passwords for authentication | Password hashing (bcrypt, argon2) | | Credentials need automatic rotation | Managing public keys only | Standard key distribution / PKI | | GitOps workflow needs encrypted secrets in repo | Small static secrets that never change | SOPS may be overkill; consider sealed secrets | ## Important Caveats - Vault's dev server mode is unencrypted and in-memory only -- NEVER use in production; always configure HA storage backend (Consul, Raft, or Integrated Storage) - AWS Secrets Manager charges per secret per month ($0.40) plus per API call -- costs can accumulate with high-frequency rotation and many microservices - Environment variables are visible in `/proc/PID/environ` on Linux, in `docker inspect`, and in CI/CD logs -- they are NOT a secure storage mechanism for production secrets - SOPS encrypts values but not keys -- file structure (key names) remain visible in plaintext, which may leak information about your architecture - Kubernetes Secrets are base64-encoded, NOT encrypted by default -- enable etcd encryption at rest or use External Secrets Operator with a real secrets manager - When a secret is compromised, rotation is not enough -- you must also revoke the old credential immediately and audit all access logs for unauthorized use - The 12-Factor App recommendation to store config in environment variables predates modern secrets managers -- for sensitive credentials, prefer mounted files or SDK-based retrieval over env vars ## Related Units - [XSS Prevention](/software/security/xss-prevention/2026) - [Docker Security Best Practices](/software/devops/docker-security/2026) - [Credential Leak Incident Response](/software/security/credential-leak-response/2026) - [Encryption Key Management (KMS)](/software/security/encryption-key-management/2026) - [Password Hashing for User Authentication](/software/security/password-hashing/2026)