Secrets Management: Best Practices and Tool Guide

TL;DR

• Bottom line: Centralize secrets in a dedicated secrets manager (Vault, AWS SM, or cloud-native equivalent), automate rotation, enforce least-privilege access, and use pre-commit hooks to prevent secrets from ever reaching version control.
• Key tool/command: vault kv get secret/myapp/db (HashiCorp Vault) or aws secretsmanager get-secret-value --secret-id myapp/db (AWS Secrets Manager)
• Watch out for: Hardcoded secrets in source code (CWE-798) -- the #1 cause of credential leaks, ranked #14 in the 2024 CWE Top 25.
• Works with: All major clouds (AWS, GCP, Azure), Kubernetes (External Secrets Operator), CI/CD pipelines, and every programming language via SDKs.

Constraints

• NEVER hardcode secrets in source code, configuration files, or container images -- CWE-798
• NEVER commit secrets to version control -- even in private repos, git history retains them permanently
• Secrets MUST be encrypted at rest and in transit -- use TLS for all secrets manager communication
• Rotate secrets automatically on a schedule appropriate to sensitivity -- API keys quarterly, database credentials monthly minimum
• Apply least-privilege access to every secret -- no blanket read-all permissions
• NEVER log secret values -- mask or redact in all application and infrastructure logs

Quick Reference

Tool Comparison

Secret Types and Recommended Storage

Decision Tree

START: What is your deployment environment?
├── Single cloud (AWS/GCP/Azure)?
│   ├── YES → Use cloud-native secrets manager (AWS SM, GCP SM, Azure KV)
│   └── NO ↓
├── Multi-cloud or hybrid?
│   ├── YES → Use HashiCorp Vault (self-hosted or HCP)
│   └── NO ↓
├── Kubernetes-only?
│   ├── YES → External Secrets Operator + cloud SM or Vault
│   └── NO ↓
├── Small team, few secrets, GitOps workflow?
│   ├── YES → SOPS + cloud KMS for encryption keys
│   └── NO ↓
├── Local development only?
│   ├── YES → dotenv (.env files in .gitignore) + git-secrets pre-commit hook
│   └── NO ↓
└── DEFAULT → HashiCorp Vault (most flexible, no lock-in)

Step-by-Step Guide

1. Install pre-commit secret detection

Prevent secrets from ever reaching version control. Install git-secrets as a pre-commit hook. [src6]

# Install git-secrets
brew install git-secrets   # macOS

# Initialize hooks in your repo
cd /path/to/your/repo
git secrets --install
git secrets --register-aws

# Add custom patterns
git secrets --add 'PRIVATE KEY'
git secrets --add 'password\s*=\s*["\047][^"\047]+'

Verify: git secrets --scan → should report any existing secrets in the repo.

2. Set up a centralized secrets manager

Choose based on your infrastructure. Example: HashiCorp Vault dev server for evaluation. [src2]

# Start Vault dev server (evaluation only)
vault server -dev -dev-root-token-id="dev-only-token"

# Store a secret
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='dev-only-token'
vault kv put secret/myapp/db username="dbadmin" password="s3cur3-p@ssw0rd"

Verify: vault kv get secret/myapp/db → should display stored key-value pairs.

3. Configure application to fetch secrets at runtime

Replace hardcoded credentials with secrets manager SDK calls. [src1]

import os, hvac  # pip install hvac>=2.4.0

client = hvac.Client(
    url=os.environ['VAULT_ADDR'],
    token=os.environ['VAULT_TOKEN']
)
secret = client.secrets.kv.v2.read_secret_version(
    path='myapp/db', mount_point='secret'
)
db_password = secret['data']['data']['password']

Verify: Application starts and connects to the database without credentials in source code.

4. Implement automated secret rotation

Configure automatic rotation to limit blast radius of compromised credentials. [src3]

# AWS: Enable rotation with Lambda
aws secretsmanager rotate-secret \
  --secret-id myapp/db-credentials \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789:function:SecretsRotation \
  --rotation-rules '{"AutomaticallyAfterDays": 30}'

Verify: aws secretsmanager describe-secret --secret-id myapp/db-credentials → RotationEnabled: true.

5. Encrypt secrets in configuration files (GitOps)

For GitOps workflows, encrypt secrets in-repo with SOPS. [src7]

# Encrypt a secrets file (values only)
sops --encrypt secrets/production.yaml > secrets/production.enc.yaml

# Decrypt at deploy time
sops --decrypt secrets/production.enc.yaml > /tmp/production.yaml

Verify: cat secrets/production.enc.yaml → keys readable, values encrypted.

Code Examples

Python: HashiCorp Vault with hvac

# Input:  VAULT_ADDR and VAULT_TOKEN environment variables
# Output: Database credentials fetched from Vault KV v2

import os
import hvac  # pip install hvac>=2.4.0

def get_vault_client():
    client = hvac.Client(
        url=os.environ['VAULT_ADDR'],
        token=os.environ['VAULT_TOKEN']
    )
    if not client.is_authenticated():
        raise RuntimeError("Vault authentication failed")
    return client

def get_db_credentials(client, path='myapp/db'):
    resp = client.secrets.kv.v2.read_secret_version(
        path=path, mount_point='secret'
    )
    return resp['data']['data']

Node.js: AWS Secrets Manager

// Input:  AWS credentials (IAM role or env vars) + secret name
// Output: Parsed secret object with database credentials

const { SecretsManagerClient, GetSecretValueCommand }
  = require("@aws-sdk/client-secrets-manager");  // ^3.500.0

async function getSecret(secretId, region = "us-east-1") {
  const client = new SecretsManagerClient({ region });
  const resp = await client.send(
    new GetSecretValueCommand({ SecretId: secretId })
  );
  return JSON.parse(resp.SecretString);
}

Python: AWS Secrets Manager with boto3

# Input:  AWS credentials (IAM role or env vars) + secret name
# Output: Parsed secret dict with credentials

import json, boto3  # pip install boto3>=1.34.0

def get_secret(secret_name, region="us-east-1"):
    client = boto3.client("secretsmanager", region_name=region)
    resp = client.get_secret_value(SecretId=secret_name)
    return json.loads(resp["SecretString"])

Bash: git-secrets Pre-Commit Hook Setup

# Input:  A git repository
# Output: Pre-commit hook that blocks secret commits

git secrets --install
git secrets --register-aws
git secrets --add 'PRIVATE KEY'
git secrets --add 'password\s*=\s*["\047][^"\047]+'

# Verify: attempt to commit a test secret
echo "AWS_SECRET=AKIAIOSFODNN7EXAMPLE" > test.txt
git add test.txt && git commit -m "test"  # blocked

Anti-Patterns

Wrong: Hardcoded secrets in source code

# BAD -- CWE-798: hardcoded credentials
import psycopg2
conn = psycopg2.connect(
    host="prod-db.example.com",
    user="admin",
    password="SuperSecret123!"  # hardcoded
)

Correct: Fetch secrets from a secrets manager

# GOOD -- credentials fetched at runtime from Vault
import os, hvac, psycopg2
client = hvac.Client(url=os.environ['VAULT_ADDR'], token=os.environ['VAULT_TOKEN'])
creds = client.secrets.kv.v2.read_secret_version(path='myapp/db')['data']['data']
conn = psycopg2.connect(
    host=creds['host'], user=creds['username'], password=creds['password']
)

Wrong: Secrets committed to git history

# BAD -- git history retains the secret forever
echo "API_KEY=sk-live-abc123xyz" > config.env
git add config.env && git commit -m "add config"
# Even git rm config.env doesn't remove from history

Correct: Pre-commit hooks + .gitignore

# GOOD -- prevent secrets from ever entering the repo
echo "*.env" >> .gitignore
git secrets --install
git secrets --register-aws

Wrong: Secrets baked into Docker image

# BAD -- secret visible via docker history
FROM node:20-alpine
ENV DATABASE_URL="postgresql://admin:secret@db:5432/myapp"
COPY . /app
CMD ["node", "server.js"]

Correct: Inject secrets at container runtime

# GOOD -- no secrets in the image
FROM node:20-alpine
COPY . /app
RUN npm install
# Run with: docker run -e DATABASE_URL="$(vault kv get ...)" myapp
CMD ["node", "server.js"]

Wrong: Env vars visible in process listing

# BAD -- visible via /proc/PID/environ
docker run -e DB_PASSWORD="secret123" myapp

Correct: Mount secrets as files in tmpfs

# GOOD -- Kubernetes: secrets via sidecar on tmpfs
apiVersion: v1
kind: Pod
spec:
  volumes:
    - name: secrets
      emptyDir:
        medium: Memory  # tmpfs
  containers:
    - name: app
      volumeMounts:
        - name: secrets
          mountPath: /mnt/secrets
          readOnly: true

Common Pitfalls

• Forgetting .gitignore before first commit: Once .env is committed, removing it from .gitignore does not remove it from git history. Fix: Add .env to .gitignore in the initial commit. If already committed, use BFG Repo-Cleaner, then rotate all exposed secrets. [src4]
• Using Parameter Store instead of Secrets Manager on AWS: AWS SSM Parameter Store lacks built-in rotation and has lower throughput limits. Fix: Use Secrets Manager for credentials needing rotation; Parameter Store for non-sensitive config. [src3]
• Single long-lived Vault token: A root or long-lived token defeats Vault's security model. Fix: Use AppRole, Kubernetes, or OIDC auth methods that issue short-lived tokens with specific policies. [src2]
• Not masking secrets in CI/CD logs: Build logs often contain expanded environment variables. Fix: Use your CI/CD platform's secret masking feature (GitHub Actions ::add-mask::, GitLab CI masked variables). [src1]
• Storing encryption keys alongside encrypted data: Defeats the purpose of encryption. Fix: Use envelope encryption -- DEK encrypted with KEK stored in separate KMS. [src1]
• Sharing secrets via Slack/email: Secrets persist in message history and backups. Fix: Use Vault's cubbyhole response wrapping or a one-time-share tool. [src1]
• No secret expiry or rotation: Static secrets accumulate risk over time. Fix: Set TTLs on all secrets; use Vault dynamic secrets for ephemeral credentials. [src2]
• Overly broad IAM policies for secrets access: Granting secretsmanager:GetSecretValue on * allows any compromised service to read all secrets. Fix: Scope policies to specific secret ARNs. [src3]

Diagnostic Commands

# Scan repo for hardcoded secrets
git secrets --scan

# Check for high-entropy strings with trufflehog
trufflehog git file://. --only-verified

# Verify Vault connectivity and authentication
vault status
vault token lookup

# List secrets in a Vault KV path
vault kv list secret/myapp/

# Check AWS Secrets Manager rotation status
aws secretsmanager describe-secret --secret-id myapp/db-credentials \
  --query '{RotationEnabled: RotationEnabled, LastRotated: LastRotatedDate}'

# Scan Docker image for embedded secrets
docker history --no-trunc myapp:latest | grep -i -E 'password|secret|key|token'

# Verify SOPS encryption
sops --decrypt secrets/production.enc.yaml > /dev/null && echo "OK"

Version History & Compatibility

When to Use / When Not to Use

Important Caveats

• Vault's dev server mode is unencrypted and in-memory only -- NEVER use in production; always configure HA storage backend (Consul, Raft, or Integrated Storage)
• AWS Secrets Manager charges per secret per month ($0.40) plus per API call -- costs can accumulate with high-frequency rotation and many microservices
• Environment variables are visible in /proc/PID/environ on Linux, in docker inspect, and in CI/CD logs -- they are NOT a secure storage mechanism for production secrets
• SOPS encrypts values but not keys -- file structure (key names) remain visible in plaintext, which may leak information about your architecture
• Kubernetes Secrets are base64-encoded, NOT encrypted by default -- enable etcd encryption at rest or use External Secrets Operator
• When a secret is compromised, rotation is not enough -- you must also revoke the old credential immediately and audit all access logs
• The 12-Factor App recommendation for env vars predates modern secrets managers -- for sensitive credentials, prefer mounted files or SDK-based retrieval