A service needs a database password, an API key, a TLS certificate. The naive approach: put secrets in environment variables, config files, or hardcode them. The problem: secrets in config files end up in git history. Environment variables get logged. Hardcoded secrets live in every build artifact. A centralized secrets management service solves this: services request secrets at runtime from a vault, which enforces access control and logs every access.

What Vault Provides#

HashiCorp Vault (or AWS Secrets Manager, GCP Secret Manager) is a service with four core capabilities:

Secret storage with encryption at rest: secrets are never stored in plaintext. Vault encrypts using AES-256-GCM with keys from a Key Management Service. Even if an attacker reads Vault’s database, they see ciphertext.

Access control policies: service A can read database/prod/password but not stripe/api-key. Access is governed by policies, not by who knows a URL.

Audit logging: every secret read, write, and deletion is logged with timestamp, caller identity, and IP. “Did service X access the payment API key before the breach?” becomes answerable.

Lease-based secrets: secrets have TTL. A service that reads a secret holds a lease. Vault can revoke all leases for a secret immediately, forcing all holders to re-authenticate.

Authentication: How Services Prove Identity#

Services can’t bootstrap with a pre-shared secret (that’s the problem we’re solving). Vault supports several auth methods:

AWS IAM auth: a service running on EC2 proves identity using its instance’s IAM role. No credentials needed: the instance metadata service provides a signed token that Vault validates against AWS APIs.

Kubernetes auth: a pod’s service account JWT token is presented to Vault. Vault validates with the Kubernetes API. The pod gets a Vault token scoped to its namespace and service account.

AppRole: a two-factor approach for non-cloud environments. A role ID (not secret, embeddable in config) plus a secret ID (secret, delivered out-of-band) together authenticate the service.

graph TD A[Service starts on Kubernetes pod] --> B[Present service account JWT to Vault auth endpoint] B --> C[Vault validates JWT against Kubernetes API] C --> D{JWT valid and policy allows?} D --> |No| E[Reject: 403 Forbidden] D --> |Yes| F[Issue Vault token with TTL and policy scope] F --> G[Service uses token to read secrets: GET /v1/database/prod/password] G --> H[Vault checks policy: is this token allowed to read this path?] H --> I[Return secret, log access: who, what, when, from where] I --> J[Service uses secret, renews Vault token before TTL expiry] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style G fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style H fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style I fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style J fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

Envelope Encryption#

Vault encrypts secrets using envelope encryption. A data encryption key (DEK) encrypts the secret. The DEK itself is encrypted using a key encryption key (KEK) stored in a hardware security module (HSM) or KMS. The KEK never leaves the HSM. To decrypt a secret, Vault asks the HSM to decrypt the DEK, then uses the DEK to decrypt the secret in memory. An attacker who steals Vault’s database gets encrypted DEKs with no HSM access to decrypt them.

Vault Seal and Unseal#

Vault starts sealed: no secrets are accessible. Unsealing loads the master key into memory, enabling operations. The master key is split using Shamir’s Secret Sharing: require 3 of 5 key shares to unseal. No single operator can unseal Vault alone. This prevents a single compromised admin from accessing all secrets.

Auto-unseal delegates unsealing to a KMS: on startup, Vault fetches its master key from AWS KMS. Removes manual intervention while preserving the HSM-backed key hierarchy.

At Oracle#

Oracle’s internal secrets management for cloud infrastructure used a combination of OCI Vault and application-level secret injection via Kubernetes init containers. Services didn’t know Vault existed: the init container fetched secrets from Vault using the pod’s service account, wrote them to a tmpfs volume, and the application read from files. Secrets never touched environment variables or config maps. When a database credential was compromised, revoking all leases in Vault and issuing a new secret took 8 minutes. Without centralized secrets management, the same rotation across 140 microservices would have required a coordinated deployment.

What I’m Learning#

The critical insight is credential bootstrapping: you can’t use a static secret to authenticate to a secrets service without recreating the problem. Cloud-native auth (AWS IAM, Kubernetes service accounts) provides a cryptographic identity that doesn’t require a pre-shared credential. Everything else builds on that foundation.

How do your services authenticate to retrieve secrets at startup, and have you had to revoke credentials rapidly in an incident?