πŸ”Vault With Kubernetes



This content originally appeared on DEV Community and was authored by Omar Ahmed

What is HashiCorp Vault?

HashiCorp Vault is a secrets & encryption platform for securely storing, generating, and controlling access to sensitive data (API keys, DB creds, TLS certs, tokens, etc.). Teams use it to centralize secrets, issue short-lived credentials on demand, and enforce least-privilege access across apps, CI/CD, and infrastructure

Install a normal Vault Without a dataStorage

kubectl create namespace vault
kubectl config set-context --current --namespace=vault
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
helm repo list
helm install vault hashicorp/vault \
  --namespace vault --create-namespace \
  --set 'server.dev.enabled=true' \
  --set 'server.dataStorage.enabled=false' \
  --set 'ui.enabled=true' 

Install a normal Vault (Raft storage, no TLS for simplicity)

cat > values.yaml <<'YAML'
server:
  dev:
    enabled: false
  standalone:
    enabled: true
    config: |
      ui = true
      listener "tcp" {
        address         = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_disable     = 1
      }
      storage "raft" {
        path = "/vault/data"
      }
  dataStorage:
    enabled: true
    size: 1Gi

ui:
  enabled: true
  serviceType: ClusterIP   # use port-forward; change to NodePort/Ingress if you prefer
YAML
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
helm -n vault install vault hashicorp/vault -f values.yaml --create-namespace --version 0.16.1
kubectl -n vault rollout status statefulset/vault

Seal/Unseal

When you start a Vault server, it starts in a sealed state. In this state, Vault can access the physical storage, but it cannot decrypt any of the data on it.
Vault encrypts the data using an encryption key (in the keyring) and stores them in its storage backend. To protect this encryption key, Vault encrypts it using another encryption key known as the root key and stores it with the data.
To decrypt the data, Vault needs the root key so that it can decrypt the encryption key. Unsealing is the process of getting access to this root key. Vault encrypts the root key using the unseal key, and stores it alongside all other Vault data.
To summarize, Vault encrypts most data using the encryption key in the keyring. To get the keyring, Vault uses the root key to decrypt it. The root key itself requires the unseal key to decrypt it.

Shamir seals

Key Shares / Unseal Keys                Root Key               Encryption Key
                                                              (protected by root key)
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Jon      🔑         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Jane     🔑         β”‚-------------->>>🔑 ---------------------->>🔑
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€                Root Key                  Encryption Key
β”‚ James    🔑         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Alison   🔑         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Pam      🔑         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Instead of distributing the unseal key to an operator as a single key, the default Vault configuration uses an algorithm known as Shamir’s Secret Sharing to split the key into shares.
Vault requires a certain threshold of shares to reconstruct the unseal key. Vault operators add shares one at a time in any order until Vault has enough shares to reconstruct the key. Then, Vault uses the unseal key to decrypt the root key. This is the Vault unseal process.
To Summarize:

  • Vault never writes the master key to disk.
  • Only encrypted data is stored (plus an encrypted data-encryption key).
  • On startup Vault can’t decrypt anything by itself, so it starts sealed.
  • Unseal = provide a quorum of unseal key shares (Shamir) to reconstruct the master key in memory β†’ that master key decrypts the data-encryption key β†’ Vault becomes unsealed and can serve requests.
❯❯❯ k get po                                                                                ⎈ (kind-my-cluster/vault)
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 0/1     Running   0          55s
vault-agent-injector-6d97459765-4mc6b   1/1     Running   0          55s

vault-0 0/1 Ready (not 1/1) >> because vault is seal

kubectl -n vault exec -it vault-0 -- vault status # sealed=true as shown below
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    0/3
Unseal Nonce       n/a
Version            1.8.3
Storage Type       raft
HA Enabled         true
❯❯❯ k exec -it vault-0 -- /bin/sh                                    ⎈ (kind-my-cluster/vault)
/ $ vault operator init 
Unseal Key 1: Q3Awl3Qw+ItIWFyLFVGFiE3OOJ1qHrHC+kQnFD2kwLbE
Unseal Key 2: OON236vYc+bf3K45J75lVHy3FmwIRdUT+mGPU3Iq4SX9
Unseal Key 3: N53iBqGPiL9RRkNNVg5DmrlD5dn6R3Vt703y6NhVUB+0
Unseal Key 4: mWPyNPitk8JTlxZByNkyYJefDo+MTpfIEcDn/lkaKuP3
Unseal Key 5: kRHm81+cYIbjO12eFJp0iJP8y3u7CR4NkOAb/4nHS5Kl

Initial Root Token: s.bQN8VBeKBga76dAUVmaGyTjJ

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 keys to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
/ $ 
/ $ vault operator unseal Q3Awl3Qw+ItIWFyLFVGFiE3OOJ1qHrHC+kQnFD2kwLbE
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true ###### it is still sealed
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       8932e24e-a03d-59e7-a694-9c9d9d6005f3
Version            1.8.3
Storage Type       raft
HA Enabled         true
/ $ vault operator unseal OON236vYc+bf3K45J75lVHy3FmwIRdUT+mGPU3Iq4SX9
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       8932e24e-a03d-59e7-a694-9c9d9d6005f3
Version            1.8.3
Storage Type       raft
HA Enabled         true
/ $ vault operator unseal N53iBqGPiL9RRkNNVg5DmrlD5dn6R3Vt703y6NhVUB+0
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false ######## it is unsealed
Total Shares            5
Threshold               3
Version                 1.8.3
Storage Type            raft
Cluster Name            vault-cluster-62a5fa0a
Cluster ID              97d206e3-d4a2-2c1f-f898-6a95c034b87e
HA Enabled              true
HA Cluster              n/a
HA Mode                 standby
Active Node Address     <none>
Raft Committed Index    24
Raft Applied Index      24
/ $ vault status
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false #<<<<<<<<<<<#########
Total Shares            5
Threshold               3
Version                 1.8.3
Storage Type            raft
Cluster Name            vault-cluster-62a5fa0a
Cluster ID              97d206e3-d4a2-2c1f-f898-6a95c034b87e
HA Enabled              true
HA Cluster              https://vault-0.vault-internal:8201
HA Mode                 active
Active Since            2025-09-03T20:26:58.922335892Z
Raft Committed Index    29
Raft Applied Index      29

Vault-0 becomes 1/1 Ready state (now vault is unsealed):

❯❯❯ k get po                                                        ⎈ (kind-my-cluster/vault)
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          27m
vault-agent-injector-6d97459765-4mc6b   1/1     Running   0          27m
/ $ vault operator init 
Unseal Key 1: Q3Awl3Qw+ItIWFyLFVGFiE3OOJ1qHrHC+kQnFD2kwLbE
Unseal Key 2: OON236vYc+bf3K45J75lVHy3FmwIRdUT+mGPU3Iq4SX9
Unseal Key 3: N53iBqGPiL9RRkNNVg5DmrlD5dn6R3Vt703y6NhVUB+0
Unseal Key 4: mWPyNPitk8JTlxZByNkyYJefDo+MTpfIEcDn/lkaKuP3
Unseal Key 5: kRHm81+cYIbjO12eFJp0iJP8y3u7CR4NkOAb/4nHS5Kl

Initial Root Token: s.bQN8VBeKBga76dAUVmaGyTjJ

/ $ vault login s.bQN8VBeKBga76dAUVmaGyTjJ
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.bQN8VBeKBga76dAUVmaGyTjJ
token_accessor       mkdn3cQnsFcO4hmklMrV8Chq
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]
/ $ 
# Enable Kubernetes Authentication
kubectl exec -n vault -it vault-0 -- vault auth enable kubernetes
# Create Service Account for App Pods
kubectl create namespace webapps
kubectl create serviceaccount vault-auth -n webapps

# Configure Kubernetes Auth in Vault
# Extract required info:
SERVICE_ACCOUNT_NAME=vault-auth
NAMESPACE=webapps

# JWT Token
TOKEN_REVIEW_JWT=$(kubectl get secret $(kubectl get serviceaccount $SERVICE_ACCOUNT_NAME -n $NAMESPACE -o jsonpath="{.secrets[0].name}") -n $NAMESPACE -o jsonpath="{.data.token}" | base64 --decode)

# Kubernetes API Host
KUBE_HOST=$(kubectl config view --raw -o=jsonpath='{.clusters[0].cluster.server}')

# Kubernetes CA Cert
KUBE_CA_CERT=$(kubectl get secret $(kubectl get serviceaccount $SERVICE_ACCOUNT_NAME -n $NAMESPACE -o jsonpath="{.secrets[0].name}") -n $NAMESPACE -o jsonpath="{.data['ca.crt']}" | base64 --decode)

# Configure in Vault:
kubectl exec -n vault -it vault-0 -- vault write auth/kubernetes/config \
    token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
    kubernetes_host="$KUBE_HOST" \
    kubernetes_ca_cert="$KUBE_CA_CERT"

Purpose of Kubernetes Auth Configuration:
This command configures Vault to trust and communicate with the Kubernetes API server so that:

  • Vault can validate that service account tokens are legitimate
  • Pods can authenticate to Vault using their Kubernetes service account tokens
  • Vault can verify which namespace and service account a request is coming from Breaking Down Each Parameter: token_reviewer_jwt
"$(kubectl get secret vault-token-reviewer -o jsonpath='{.data.token}' | base64 -d)"
  • What it is: A service account token that Vault uses to authenticate with the Kubernetes API
  • Why needed: Vault needs to call the Kubernetes TokenReview API to validate incoming service account tokens
  • What it does: Acts as Vault’s “identity” when talking to Kubernetes

kubernetes_host

"https://kubernetes.default.svc:443"
  • What it is: The URL of the Kubernetes API server from inside the cluster
  • Why needed: Vault needs to know where to send TokenReview requests
  • kubernetes.default.svc: This is the internal DNS name for the Kubernetes API server

kubernetes_ca_cert

"$(kubectl get secret vault-token-reviewer -o jsonpath='{.data.ca\.crt}' | base64 -d)"
  • What it is: The Certificate Authority certificate for the Kubernetes cluster
  • Why needed: Vault uses this to verify the SSL certificate when connecting to the Kubernetes API
  • Security: Prevents man-in-the-middle attacks

The Authentication Flow:
Here’s what happens when a pod tries to authenticate to Vault:

1. Pod β†’ Vault: "Here's my service account token"
2. Vault β†’ Kubernetes API: "Is this token valid?" (using token_reviewer_jwt)
3. Kubernetes API β†’ Vault: "Yes, it belongs to service account X in namespace Y"
4. Vault β†’ Pod: "Authentication successful, here's your Vault token"

Without this configuration:

  • Vault can’t verify that service account tokens are real
  • Any pod could potentially forge authentication requests
  • You lose the security boundary that Kubernetes provides

With this configuration:

  • Vault cryptographically verifies each authentication request
  • Only legitimate pods with valid service account tokens can authenticate
  • You get fine-grained access control based on Kubernetes RBAC

This step essentially creates a trust relationship between Vault and your Kubernetes cluster, enabling secure, automated authentication for your workloads.

Create Vault Policy: Create a file myapp-policy.hcl:
A policy in Vault is a set of rules (paths + capabilities) that defines what a token/identity is allowed to do.
Vault is deny-by-defaultβ€”nothing is allowed unless a policy explicitly grants it.

# Access to read/write secret data
path "secret/data/mysql" {  
  capabilities = ["create", "update", "read", "delete", "list"]
}

path "secret/data/frontend" {
  capabilities = ["create", "update", "read", "delete", "list"]
}

# Access to list secrets under the path
path "secret/metadata/mysql" {
  capabilities = ["list"]
}

path "secret/metadata/frontend" {
  capabilities = ["list"]
}

Apply Vault Policy:

kubectl cp myapp-policy.hcl vault/vault-0:/tmp/myapp-policy.hcl
kubectl exec -n vault -it vault-0 -- vault policy write myapp-policy /tmp/myapp-policy.hcl
vault policy read <policy name>                   # show a policy

Create Role in Vault to Map Pod to Policy:

kubectl exec -n vault -it vault-0 -- vault write auth/kubernetes/role/vault-role \
    bound_service_account_names=vault-auth \
    bound_service_account_namespaces="webapps" \
    policies=myapp-policy \
    ttl=24h

Store Secrets in Vault:

# Enable KV V2 Engine
kubectl exec -n vault -it vault-0 -- vault secrets enable -path=secret -version=2 kv
# Store Secrets in Vault
kubectl exec -n vault -it vault-0 -- vault kv put secret/mysql MYSQL_DATABASE=bankappdb MYSQL_ROOT_PASSWORD=Test@123
kubectl exec -n vault -it vault-0 -- vault kv put secret/frontend MYSQL_ROOT_PASSWORD=Test@123

Create YAML Manifest File (With Below Configurations):
Example annotation block for a pod:

annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/role: "vault-role"
  vault.hashicorp.com/agent-inject-secret-MYSQL_ROOT_PASSWORD: "secret/mysql"
  vault.hashicorp.com/agent-inject-template-MYSQL_ROOT_PASSWORD: |
    {{- with secret "secret/mysql" -}}
    export MYSQL_ROOT_PASSWORD="{{ .Data.data.MYSQL_ROOT_PASSWORD }}"
    {{- end }}

what to read and how to write.
vault.hashicorp.com/agent-inject-secret-MYSQL_ROOT_PASSWORD: "<vault path>"
Tells the injector which secret to fetch from Vault and that it should write a file named MYSQL_ROOT_PASSWORD under /vault/secrets/ (unless you override the filename/path).
vault.hashicorp.com/agent-inject-template-MYSQL_ROOT_PASSWORD:
Provides a custom template for what to write into that file. Without this, the injector writes a default rendering (key=value lines). With it, you control the exact content.
Template: It renders a shell line into the injected file, like: export MYSQL_ROOT_PASSWORD="supersecret123"

    {{- with secret "secret/mysql" -}}
    export MYSQL_ROOT_PASSWORD="{{ .Data.data.MYSQL_ROOT_PASSWORD }}"
    {{- end }}

The sidecar Vault Agent will:
β€’ Auth using the service account token
β€’ Fetch secrets from Vault
β€’ Write them to /vault/secrets/… >>> /vault/secrets/MYSQL_ROOT_PASSWORD inside the pod

UI access

kubectl -n vault get svc vault
kubectl -n vault port-forward --address=0.0.0.0 svc/vault 8200:8200

Clean-up steps

helm uninstall vault
kubectl -n vault delete pod -l app.kubernetes.io/name=vault --force --grace-period=0 || true
kubectl -n vault delete pvc -l app.kubernetes.io/name=vault || true


This content originally appeared on DEV Community and was authored by Omar Ahmed