Secrets Lifecycle

Generation, distribution, rotation, revocation, audit. Secrets are not a one-time configuration — they have a lifecycle. The pattern under Vault + ESO + sealed-secrets + cloud KMS.

Secrets are living — they get generated, distributed, rotated, revoked, audited. Treat them as configuration and they age into liability. Status: STUB — promoted to OUTLINE in Y3 Phase 27.

What this pattern is

The secrets lifecycle treats secrets (API keys, database passwords, certificates, tokens) as living artifacts with operational stages: generation (machine-generated, high-entropy, never typed by humans); distribution (encrypted-at-rest in storage, encrypted-in-transit to consumers, never in logs or environment-variable dumps); rotation (scheduled or event-triggered renewal, with old secret remaining valid briefly during cut-over); revocation (compromised or replaced secret invalidated immediately); audit (every access logged with identity plus time plus reason). Tools: HashiCorp Vault plus External Secrets Operator (the K8s-native combination /root uses), sealed-secrets (for GitOps-friendly secret distribution), and cloud KMS (AWS KMS, GCP KMS) for envelope encryption.

The lifecycle framing matters because static secrets are a security debt that compounds. A key leaked in a Git commit five years ago is still in the history; a service-account token without rotation is still valid; a long-lived database password is a perpetual blast-radius. Every incident report from the past decade includes a version of “a credential exposed years ago was still valid at the time of the breach.” Rotating every secret every 90 days at minimum bounds the blast radius of every past exposure to a 90-day window.

Well-designed secrets tooling makes the lifecycle nearly free. Vault generates dynamic database credentials on demand with short TTLs; External Secrets Operator syncs them into Kubernetes; workloads consume them through standard Secret mounts; rotation is a Vault configuration, not a per-service change. What was manual rotation in the 2010s (“update the config file on every server”) became automatic in the 2020s (Vault + K8s + ESO handle everything).

Concrete instances in the wild

  • HashiCorp Vault + External Secrets Operator on basecamp. Vault holds long-lived root secrets and generates short-lived per-caller secrets. ESO syncs them into Kubernetes Secrets; workloads mount those Secrets; rotation is transparent.
  • AWS Secrets Manager + rotation Lambdas. Managed service that rotates database credentials by invoking a Lambda function. Native integration with RDS, DocumentDB, Redshift.
  • Google Secret Manager with automatic versioning. Every rotation creates a new version; consumers can pin to latest or specific versions. Old versions kept for rollback.
  • Bitnami sealed-secrets. GitOps-friendly: encrypt secrets client-side, commit the encrypted form to Git, cluster-side controller decrypts at admission. No secret manager required.
  • Cloud KMS envelope encryption. Long-lived data-encryption keys wrapped by short-lived key-encryption keys, both managed by the cloud KMS. Rotation of the outer key doesn’t require re-encrypting data.
  • Kubernetes ServiceAccount token projection. Short-lived, audience-bound tokens generated per pod. The TokenRequest API replaced permanent tokens in K8s 1.24+.
  • AWS IAM Roles for Service Accounts (IRSA). Kubernetes pods assume AWS IAM roles via OIDC federation. No AWS credentials stored anywhere; tokens issued dynamically.
  • Certificate authorities with short TTLs. cert-manager + Let’s Encrypt or a private CA issues certificates with 90-day (or 24-hour) TTLs and automatic renewal.
  • SSH certificate authorities. Instead of long-lived SSH keys distributed manually, an SSH CA issues short-lived certificates. Vault, HashiCorp Boundary, and Teleport all implement this pattern.

Why this pattern matters

Every serious credential-related breach in the last decade shares a shape: a credential that shouldn’t have been valid was still valid when the attacker used it. GitHub’s 2022 pypi.org credential leak in Azure: a Personal Access Token in a build artifact from 2020 was still valid in 2022. The Uber breach: an MFA-fatigued engineer approved a login request from an attacker with a valid credential dump from an earlier breach. In both cases, disciplined rotation would have bounded the exposure to a much shorter window.

The pattern’s operational value also compounds. Teams that rotate everything regularly discover their rotation tooling; when a real incident requires emergency revocation, the tooling is already tested. Teams that never rotate discover during incidents that rotation is manual, slow, and error-prone, exactly when they need it to be fast.

Modern secrets tooling makes the pattern effectively free. Vault’s dynamic secrets means database passwords are generated per-caller with 30-minute TTLs; nobody has to rotate manually because everything rotates continuously. Cloud KMS’s automatic key rotation means encryption keys renew every 90 days without operator intervention. Kubernetes ServiceAccount token projection means every pod gets a fresh audience-bound token per session. What used to require dedicated tooling and dedicated engineers now happens by default in properly-configured platforms.

The counter-failure mode is over-engineering. Teams that try to rotate everything every hour discover downstream systems that don’t handle rotation gracefully (long-lived DB connections, cached credentials, etc.). Balance rotation frequency with downstream tolerance. Ninety-day rotation is aggressive-but-realistic for most workloads; 30 minutes for dynamic secrets that never leave process memory.

Depth progression

STUB     ← you are here.
OUTLINE  Promoted when Y3 Phase 27 deploys Vault + ESO on basecamp.
DEEP     Promoted after Y3 end — at least one successful rotation event AND
         one successful revocation event observed on basecamp.

Preview: what OUTLINE will answer

When Y3 Phase 27 promotes this entry to OUTLINE, it will name:

  • PROBLEM. How do you keep secrets under continuous management so that past exposures have bounded blast radius?
  • PRINCIPLES. Every secret has an expiration. Rotation is automated. Revocation is one API call. Audit trail for every access. Distribution goes through a managed system, never through Git or environment-variable dumps. Dynamic secrets preferred over static.
  • TRADE-OFFS. Rotation frequency (aggressive = smaller blast radius, more churn) vs downstream tolerance. Dynamic secrets (per-caller, short-lived) vs static (per-service, long-lived). Centralized secret store (Vault, cloud secret manager) vs distributed (sealed-secrets in Git). Envelope encryption vs single-key.
  • TOOLS (time-stamped as of 2026-06): Vault + External Secrets Operator (basecamp reference), AWS Secrets Manager, Google Secret Manager, Azure Key Vault, sealed-secrets (GitOps), cloud KMS (envelope encryption), cert-manager (certificates), Teleport/Boundary (SSH cert authorities).

The DEEP promotion, after Y3 end with at least one rotation and revocation event, will add MASTERY (operating Vault + ESO through routine rotations), COMPARE (Vault vs cloud secret managers vs sealed-secrets as three different distribution models), OPERATE (a real rotation or revocation event, documented in ops-handbook), and CONTRIBUTE (a Vault or ESO documentation improvement or plugin contribution).

Canonical references

  • HashiCorp Vault documentation, particularly the dynamic secrets and secret engines sections. Free at hashicorp.com.
  • External Secrets Operator documentation and reference architecture patterns.
  • Google’s SRE Book, chapter on production credentials — background on the discipline at hyperscale.
  • NIST SP 800-57, Recommendation for Key Management — the authoritative reference for cryptographic key lifecycle.
  • Kubernetes documentation on ServiceAccount token volumes and TokenRequest API.

Cross-references