Accessing HashiCorp Vault Secrets from Tanzu Application Platform Workloads

Views JA MD
Kubernetes Cartographer Tanzu TAP Vault

⚠️ This article was automatically translated by OpenAI (gpt-4o). It may be edited eventually, but please be aware that it may contain incorrect information at this time.

In Tanzu Application Platform, it is common to configure sensitive information for Workloads by setting K8s Secrets via Service Binding. However, if you do not want to place sensitive information in K8s Secrets, you might want to use HashiCorp Vault.

To access Vault from K8s, you can use:

However,

  • It is difficult to configure CSI Volume for TAP Workloads without hacks.
  • External Secrets syncs Vault Secrets to K8s Secrets, which does not meet the need to avoid placing sensitive information in K8s Secrets.

Therefore, we will use Vault Agent Injector.

External Secret is experimentally supported in Tanzu Application Platform 1.4.
https://docs.vmware.com/en/VMware-Tanzu-Application-Platform/1.4/tap/external-secrets-about-external-secrets-operator.html

This time, we will deploy a Workload on TAP on Kind, created in this article, and refer to the Secret set on Vault.

In TAP, a Multi Cluster configuration is common, and Vault is likely to run in a different location from the Run Cluster where the Workload resides. Therefore, we will access an existing Vault from the Workload. We referred to the following tutorial.

https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-external-vault

Table of Contents

Starting Vault

Start Vault outside the K8s cluster. This time, we will start it in dev mode.

vault server -dev -dev-root-token-id root -dev-listen-address 0.0.0.0:8200

Installing Vault Agent Injector

Install Vault Agent Injector on the cluster where TAP is installed (Run Cluster in a Multi Cluster configuration).

Use the Helm Chart.

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

Generate the manifest with helm template. Specify the Vault URL as seen from the K8s cluster in injector.externalVaultAddr.

helm template vault hashicorp/vault -n vault --set "injector.externalVaultAddr=http://host.docker.internal:8200" > vault-agent-injector.yaml
kubectl create ns vault
kubectl apply -f vault-agent-injector.yaml -n vault

Check the Pod.

$ kubectl get pod -n vault
NAME                                    READY   STATUS    RESTARTS   AGE
vault-agent-injector-547d8fc8db-kxtd7   1/1     Running   0          14s

Enabling Kubernetes Auth Method

Enable the Kubernetes Auth Method.

vault auth enable kubernetes

Create a token for the Service Account.

kubectl apply -n vault -f -<<EOF
apiVersion: v1
kind: Secret
metadata:
  name: vault-token
  namespace: vault
  annotations:
    kubernetes.io/service-account.name: vault
type: kubernetes.io/service-account-token
EOF

Set the K8s information to access Vault.

TOKEN_REVIEW_JWT=$(kubectl get secret -n vault vault-token -otemplate='{{index .data "token" | base64decode}}')
KUBE_CA_CERT=$(kubectl get secret -n vault vault-token -otemplate='{{index .data "ca.crt"| base64decode}}')
KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')

vault write auth/kubernetes/config \
     token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
     kubernetes_host="$KUBE_HOST" \
     kubernetes_ca_cert="$KUBE_CA_CERT" \
     issuer="https://kubernetes.default.svc.cluster.local"

Registering Secrets for the Application

We will use https://github.com/making/vehicle-api for the application. This app is a simple Spring Boot application that accesses PostgreSQL and is completely independent of Vault.
We will use the free plan of ElephantSQL for the PostgreSQL to be accessed.

Create an instance on ElephantSQL and register the instance information as a Secret for vehicle-api.

vault kv put secret/vehicle-api/config \
  host='floppy.db.elephantsql.com' \
  username='ixyepwbw' \
  password='QIDpi7cBKioNGyp7JeN8ZMTL-rIWL_9B' \
  database='ixyepwbw'

Check the registered information.

$ vault read secret/data/vehicle-api/config
Key         Value
---         -----
data        map[database:ixyepwbw host:floppy.db.elephantsql.com password:QIDpi7cBKioNGyp7JeN8ZMTL-rIWL_9B username:ixyepwbw]
metadata    map[created_time:2023-01-11T08:46:51.471224Z custom_metadata:<nil> deletion_time: destroyed:false version:1]

Create a read-only Policy for this Secret.

vault policy write vehicle-api - <<EOF
path "secret/data/vehicle-api/config" {
  capabilities = ["read"]
}
EOF

Deploying the Workload (Adding application.properties)

Create a Role for the default Service Account in the demo namespace to deploy the Workload and attach the previously created Policy.

vault write auth/kubernetes/role/vehicle-api \
     bound_service_account_names=default \
     bound_service_account_namespaces=demo \
     policies=vehicle-api \
     ttl=24h

Set annotations for the Workload to access this Secret.

Write the contents of the Secret to /vault/secrets/application.properties and set the environment variable SPRING_CONFIG_IMPORT to read this file from the app.

cat <<EOF > workload.yaml
apiVersion: carto.run/v1alpha1
kind: Workload
metadata:
  labels:
    app.kubernetes.io/part-of: vehicle-api
    apps.tanzu.vmware.com/workload-type: web
  name: vehicle-api
spec:
  params:
  - name: annotations
    value:
      autoscaling.knative.dev/minScale: '1'
      vault.hashicorp.com/agent-inject: 'true'
      vault.hashicorp.com/role: vehicle-api
      vault.hashicorp.com/agent-inject-secret-application.properties: secret/data/vehicle-api/config
      vault.hashicorp.com/agent-inject-template-application.properties: |
        {{- with secret "secret/data/vehicle-api/config" -}}
        spring.datasource.url=jdbc:postgresql://{{ .Data.data.host }}/{{ .Data.data.database }}
        spring.datasource.username={{ .Data.data.username }}
        spring.datasource.password={{ .Data.data.password }}
        {{- end -}}
  env:
  - name: SPRING_CONFIG_IMPORT
    value: /vault/secrets/application.properties
  build:
    env:
    - name: BP_JVM_VERSION
      value: "17"
  source:
    git:
      url: https://github.com/making/vehicle-api
      ref:
        branch: main
EOF

tanzu apps workload apply -f workload.yaml -n demo
# or kubectl apply -f workload.yaml -n demo

vault.hashicorp.com/agent-inject-template-... could not be set with the tanzu CLI --annotation.
If creating the Workload with CLI only, the following parameter specification is required.

tanzu apps workload apply vehicle-api2 \
  -n demo\
  --git-repo https://github.com/making/vehicle-api \
  --git-branch main \
  --type web \
  --app vehicle-api \
  --env SPRING_CONFIG_IMPORT=/vault/secrets/application.properties \
  --build-env BP_JVM_VERSION=17 \
  --param-yaml annotation='{"autoscaling.knative.dev/minScale":"1","vault.hashicorp.com/agent-inject":"true","vault.hashicorp.com/agent-inject-secret-application.properties":"secret/data/vehicle-api/config","vault.hashicorp.com/agent-inject-template-application.properties":"{{- with secret \"secret/data/vehicle-api/config\" -}}\nspring.datasource.url=jdbc:postgresql://{{ .Data.data.host }}/{{ .Data.data.database }}\nspring.datasource.username={{ .Data.data.username }}\nspring.datasource.password={{ .Data.data.password }}\n{{- end -}}","vault.hashicorp.com/role":"vehicle-api"}'

After the app is deployed, check the Workload. The Pod will include a vault agent sidecar, so the number of containers will be 3/3.

 $ tanzu apps workload get -n demo vehicle-api
📡 Overview
   name:   vehicle-api
   type:   web

💾 Source
   type:     git
   url:      https://github.com/making/vehicle-api
   branch:   main

📦 Supply Chain
   name:   source-to-url

   RESOURCE           READY   HEALTHY   TIME   OUTPUT
   source-provider    True    True      134m   GitRepository/vehicle-api
   image-provider     True    True      131m   Image/vehicle-api
   config-provider    True    True      131m   PodIntent/vehicle-api
   app-config         True    True      131m   ConfigMap/vehicle-api
   service-bindings   True    True      131m   ConfigMap/vehicle-api-with-claims
   api-descriptors    True    True      131m   ConfigMap/vehicle-api-with-api-descriptors
   config-writer      True    True      131m   Runnable/vehicle-api-config-writer

🚚 Delivery
   name:   delivery-basic

   RESOURCE          READY   HEALTHY   TIME   OUTPUT
   source-provider   True    True      130m   ImageRepository/vehicle-api-delivery
   deployer          True    True      130m   App/vehicle-api

💬 Messages
   No messages found.

🛶 Pods
   NAME                                            READY   STATUS      RESTARTS   AGE
   vehicle-api-00001-deployment-7b6d9f7f49-rl25z   3/3     Running     0          30m
   vehicle-api-build-1-build-pod                   0/1     Completed   0          34m
   vehicle-api-config-writer-trr8r-pod             0/1     Completed   0          31m

🚢 Knative Services
   NAME          READY   URL
   vehicle-api   Ready   https://vehicle-api-demo.127-0-0-1.sslip.io

To see logs: "tanzu apps workload tail vehicle-api --namespace demo"

Access the app and confirm that data can be retrieved from the database.

$ curl -sk https://vehicle-api-demo.127-0-0-1.sslip.io/vehicles
[{"id":1,"name":"Avalon"},{"id":2,"name":"Corolla"},{"id":3,"name":"Crown"},{"id":4,"name":"Levin"},{"id":5,"name":"Yaris"},{"id":6,"name":"Vios"},{"id":7,"name":"Glanza"},{"id":8,"name":"Aygo"}]

Confirm that the Secret obtained from Vault is written to the file.

POD_NAME=$(kubectl get pod -n demo -l serving.knative.dev/service=vehicle-api -ojsonpath='{.items[0].metadata.name}')

$ kubectl exec -it ${POD_NAME} -n demo -c workload -- ls -la /vault/secrets/
total 8
drwxrwxrwt 2 root root   60 Jan 11 09:14 .
drwxr-xr-x 3 root root 4096 Jan 11 09:14 ..
-rw-r--r-- 1 _apt cnb   170 Jan 11 09:14 application.properties

$ kubectl exec -it ${POD_NAME} -n demo -c workload -- cat /vault/secrets/application.properties
spring.datasource.url=jdbc:postgresql://floppy.db.elephantsql.com/ixyepwbw
spring.datasource.username=ixyepwbw
spring.datasource.password=QIDpi7cBKioNGyp7JeN8ZMTL-rIWL_9

Access the Actuator env endpoint and confirm that the settings are actually read from this file.

kubectl port-forward ${POD_NAME} -n demo 8081:8081

$ curl -s localhost:8081/actuator/env | jq .
{
  "activeProfiles": [],
  "propertySources": [
    {
      "name": "server.ports",
      "properties": {
        ...
      }
    },
    {
      "name": "servletContextInitParams",
      "properties": {}
    },
    {
      "name": "systemProperties",
      "properties": {
        ...
      }
    },
    {
      "name": "systemEnvironment",
      "properties": {
        ...
      }
    },
    {
      "name": "Config resource 'file [/vault/secrets/application.properties]' via location '/vault/secrets/application.properties'",
      "properties": {
        "spring.datasource.url": {
          "value": "jdbc:postgresql://floppy.db.elephantsql.com/ixyepwbw",
          "origin": "URL [file:/vault/secrets/application.properties] - 1:23"
        },
        "spring.datasource.username": {
          "value": "ixyepwbw",
          "origin": "URL [file:/vault/secrets/application.properties] - 2:28"
        },
        "spring.datasource.password": {
          "value": "******",
          "origin": "URL [file:/vault/secrets/application.properties] - 3:28"
        }
      }
    },
    {
      "name": "Config resource 'class path resource [application.properties]' via location 'optional:classpath:/'",
      "properties": {
        ...
      }
    }
  ]
}

We successfully accessed Vault from TAP. It was confirmed that sensitive information could be retrieved from TAP without using K8s Secrets.