Secrets Management Guide

Overview

This guide covers two main approaches for managing secrets in Kubernetes:

1. External Secrets Operator (ESO)
2. Secrets Store CSI Driver

Prerequisites

Kubernetes Cluster Requirements

• Kubernetes 1.19+
• RBAC enabled
• cert-manager installed (required for ESO webhook)
• Secrets Store CSI Driver installed

Cloud Provider Requirements

AWS

• AWS CLI installed and configured
• IAM roles with necessary permissions:
  • secretsmanager:GetSecretValue
  • secretsmanager:DescribeSecret
  • secretsmanager:ListSecrets
• eksctl for EKS clusters

Azure

• Azure CLI installed and configured
• Azure subscription with Key Vault access
• Service Principal with required permissions:
  • Key Vault Secrets User
  • Key Vault Secrets Officer

GCP

• Google Cloud SDK installed and configured
• Service account with Secret Manager access
• Workload Identity configured

Required Tools

# Helm 3.0+
curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

# kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

External Secrets Operator (ESO)

Installation

1. Add the Helm repository:

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

2. Install ESO:

helm install external-secrets \
  external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace \
  --set installCRDs=true

Cloud Provider Setup

AWS Secrets Manager

1. Create IAM policy:

aws iam create-policy \
  --policy-name external-secrets-policy \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "secretsmanager:GetSecretValue",
          "secretsmanager:DescribeSecret"
        ],
        "Resource": "*"
      }
    ]
  }'

2. Create SecretStore:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-backend
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-west-2
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa

Azure Key Vault

1. Create Key Vault:

az keyvault create \
  --name prime-edm-kv \
  --resource-group prime-edm-rg \
  --location eastus

2. Create SecretStore:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: azure-backend
spec:
  provider:
    azurekv:
      tenantId: "your-tenant-id"
      vaultUrl: "https://prime-edm-kv.vault.azure.net"
      authSecretRef:
        clientId:
          name: azure-secret-creds
          key: client-id
        clientSecret:
          name: azure-secret-creds
          key: client-secret

Google Secret Manager

1. Enable Secret Manager API:

gcloud services enable secretmanager.googleapis.com

2. Create SecretStore:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: gcp-backend
spec:
  provider:
    gcpsm:
      projectID: your-project-id
      auth:
        workloadIdentity:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: default

Creating External Secrets

1. Create an ExternalSecret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-secret
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-backend
    kind: SecretStore
  target:
    name: db-credentials
  data:
  - secretKey: username
    remoteRef:
      key: prod/db/credentials
      property: username
  - secretKey: password
    remoteRef:
      key: prod/db/credentials
      property: password

Secrets Store CSI Driver

Installation

1. Add the Secrets Store CSI Driver Helm repository:

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm repo update

2. Install the driver:

helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \
  --namespace kube-system \
  --set syncSecret.enabled=true \
  --set enableSecretRotation=true

Cloud Provider Setup

AWS Provider

1. Install the AWS provider:

helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws
helm repo update

helm install secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws \
  --namespace kube-system

2. Create SecretProviderClass:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "prod/db/credentials"
        objectType: "secretsmanager"
        objectAlias: "DB_CREDENTIALS"
  secretObjects:
    - secretName: db-credentials
      type: Opaque
      data:
        - objectName: DB_CREDENTIALS
          key: username
        - objectName: DB_CREDENTIALS
          key: password

Azure Provider

1. Install the Azure provider:

helm repo add csi-secrets-store-provider-azure https://azure.github.io/secrets-store-csi-driver-provider-azure/charts
helm repo update

helm install azure-csi-provider csi-secrets-store-provider-azure/csi-secrets-store-provider-azure \
  --namespace kube-system

2. Create SecretProviderClass:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-secrets
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "<client-id>"
    keyvaultName: "<key-vault-name>"
    objects: |
      array:
        - |
          objectName: dbUsername
          objectType: secret
        - |
          objectName: dbPassword
          objectType: secret
    tenantId: "<tenant-id>"

GCP Provider

1. Install the GCP provider:

helm repo add secrets-store-csi-driver-provider-gcp https://kubernetes-sigs.github.io/secrets-store-csi-driver-provider-gcp/charts
helm repo update

helm install secrets-provider-gcp secrets-store-csi-driver-provider-gcp/secrets-store-csi-driver-provider-gcp \
  --namespace kube-system

2. Create SecretProviderClass:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: gcp-secrets
spec:
  provider: gcp
  parameters:
    secrets: |
      - resourceName: "projects/$PROJECT_ID/secrets/db-username/versions/latest"
        path: "username"
      - resourceName: "projects/$PROJECT_ID/secrets/db-password/versions/latest"
        path: "password"

Using Secrets in Pods

1. Mount secrets as volumes:

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  volumes:
    - name: secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "aws-secrets"  # or azure-secrets, gcp-secrets
  containers:
    - name: app
      image: app:latest
      volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
      env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password

Best Practices

1. Enable Secret Auto-Rotation:

spec:
  volumes:
    - name: secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        volumeAttributes:
          secretProviderClass: "aws-secrets"
          refreshInterval: "1h"  # Adjust as needed

2. Use Pod Identity:

   • AWS: Use IAM Roles for Service Accounts (IRSA)
   • Azure: Use Managed Identities
   • GCP: Use Workload Identity

3. Implement Health Checks:

spec:
  containers:
    - name: app
      livenessProbe:
        exec:
          command:
            - cat
            - /mnt/secrets-store/username
        initialDelaySeconds: 5
        periodSeconds: 30

Troubleshooting

1. Check provider pods:

kubectl get pods -n kube-system -l app=secrets-store-csi-driver

2. View provider logs:

kubectl logs -n kube-system -l app=secrets-store-csi-driver

3. Verify SecretProviderClass:

kubectl describe secretproviderclass aws-secrets

4. Check mounted secrets:

kubectl exec -it app -- ls -l /mnt/secrets-store

Comparison: ESO vs CSI Driver

External Secrets Operator

• Pros:
  • Kubernetes-native secret management
  • Automatic secret rotation
  • Support for multiple providers
  • No pod restarts needed for updates
• Cons:
  • Additional operator to maintain
  • Requires CRDs

Secrets Store CSI Driver

• Pros:
  • Direct volume mounting
  • Native integration with cloud providers
  • Support for certificates
• Cons:
  • Requires pod restarts for updates
  • More complex setup
  • Limited to volume mounts

Security Best Practices

1. Secret Rotation

   • Enable automatic rotation
   • Set appropriate refresh intervals
   • Monitor rotation status

2. Access Control

   • Use RBAC strictly
   • Implement least privilege
   • Regular access reviews

3. Monitoring

   • Set up alerts for failures
   • Monitor secret access
   • Audit logging

4. Backup and Recovery

   • Regular backups of configurations
   • Document recovery procedures
   • Test restoration process