---
title: Install Tanzu Application Platform with GitOps by kapp-controller
tags: ["Kubernetes", "Cartographer", "kind", "Tanzu", "TAP", "Knative", "kapp-controller", "ytt", "GitOps"]
categories: ["Dev", "CaaS", "Kubernetes", "TAP"]
date: 2022-12-17T05:11:58Z
updated: 2022-12-19T04:08:37Z
---

Install TAP with GitOps. GitOps allows you to install TAP itself declaratively, manage configuration files, and prevent cluster operation errors. There are several ways to achieve GitOps in Kubernetes, here we use [kapp-controller](https://carvel.dev/kapp-controller/). kapp-controller is a prerequisite for installing TAP, so you don't need to install any additional software for GitOps.

I talked abount kapp-controller at Cloud Native Operator Days Tokyo 2022, so please refer to the [slides (in Japanese)](https://docs.google.com/presentation/d/1mxsONks4IoQqEQ37YBPNS4hkKa71ScfsTHGt1odyMVc=) or [video (in Japanese)](https://www.youtube.com/watch?v=mRAKLIZ4bqs) for details.

This time it is like the GitOps version of installing TAP manually in [this article](https://ik.am/entries/716).


**table of contents**
<!-- toc -->

### Create a kind cluster
Use [kind](https://kind.sigs.k8s.io/) as the environment to install TAP .

Allocate at least 4 CPUs and 4 GB memory to Docker.


The following version is used. K8s 1.24 node didn't start on the old kind cluster.

```
$ kind version
kind v0.17.0 go1.19.2 darwin/amd64
```


Use k8s 1.24.


```
cat <<EOF > kind-expose-port.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
 - role: control-plane
   extraPortMappings:
   - containerPort: 31443 # expose port 31443 of the node to port 80 on the host for use later by Contour ingress (envoy)
     hostPort: 443
   - containerPort: 31080 # expose port 31080 of the node to port 80 on the host for use later by Contour ingress (envoy)
     hostPort: 80
EOF

kind create cluster --config kind-expose-port.yaml --image kindest/node:v1.24.7
```

### Install kapp-controller

The TAP document installs kapp-controller with [Cluster Essentials for VMware Tanzu](https://network.tanzu.vmware.com/products/tanzu-cluster-essentials/), but here we use the manifest on Github to install kapp-controller with the minimum number of steps.

```
kubectl apply -f https://github.com/vmware-tanzu/carvel-kapp-controller/releases/download/v0.44.1/release.yml
```

### Create a Service Account

Create a Service Account for use by the kapp controller and set the ClusterRoleBinding. Here we bind the `cluster-admin`, but you can also use a restricted ClusterRole.


```
NAMESPACE=kapp
kubectl create ns ${NAMESPACE}
kubectl create -n ${NAMESPACE} sa kapp
kubectl create clusterrolebinding kapp-cluster-admin-${NAMESPACE} --clusterrole cluster-admin --serviceaccount=${NAMESPACE}:kapp
```

### Create directories for GitOps

`kapp` directory contains the kapp-controller's App CR definition, and `config` directory contains the k8s resource definition to be deployed. 
The files are placed per cluster that creates it. This time, we will create `kind-iterate` directory to manage the iterator cluster deployed on kind. These will be pushed to a git repo later.

```
mkdir -p tap-install-gitops/kind-iterate/kapp
mkdir -p tap-install-gitops/kind-iterate/config
```

The file hierarchy will look like bellow:

```
$ tree tap-install-gitops 
tap-install-gitops
`-- kind-iterate
    |-- config
    `-- kapp
```

Manifests will be stored in https://github.com/making/tap-install-gitops .


### Define App CRs

Define the kapp-controller's [App CR](https://carvel.dev/kapp-controller/docs/v0.43.2/app-overview/) and the resources deployed from the App CR.

#### App CR for secretgen-controller installation

First , define the installation of [secretgen-controller](https://github.com/vmware-tanzu/carvel-secretgen-controller) , which is another prerequisite for TAP, in App CR. 
Here we will deploy the secretgen-controller from the manifest on Github in stead of Cluster Essentials.


```yaml
cat <<EOF > tap-install-gitops/kind-iterate/kapp/secretgen-controller.yaml
apiVersion: kappctrl.k14s.io/v1alpha1
kind: App
metadata:
  name: secretgen-controller
  namespace: kapp
  annotations:
    kapp.k14s.io/change-group: "{name}"
spec:
  serviceAccountName: kapp
  fetch:
  - http:
      url: https://github.com/vmware-tanzu/carvel-secretgen-controller/releases/download/v0.13.0/release.yml
  syncPeriod: 6h
  template:
  - ytt: { }
  deploy:
  - kapp:
      rawOptions:
      - --wait-timeout=5m
      - --diff-changes=true
      - --diff-mask=true
EOF
```

The file hierarchy will look like bellow:

```
$ tree tap-install-gitops 
tap-install-gitops
`-- kind-iterate
    |-- config
    `-- kapp
        `-- secretgen-controller.yaml
```

#### App CR for TAP's PackageRepository

Define PackageRepository and Namespace for TAP and the Secret required to use it in App CRs.

```
mkdir -p tap-install-gitops/kind-iterate/config/tap-repository
```

Namespace definition

```yaml
cat <<EOF > tap-install-gitops/kind-iterate/config/tap-repository/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: tap-install
EOF
```

Definition of Secrets. Registry information is externalized and passed from the outside later.

```yaml
cat <<EOF > tap-install-gitops/kind-iterate/config/tap-repository/secret.yaml
#@ load("@ytt:data", "data")
apiVersion: v1
kind: Secret
metadata:
  name: tap-registry
  namespace: tap-install
type: kubernetes.io/dockerconfigjson
stringData:
  #@yaml/text-templated-strings
  .dockerconfigjson: |-
    {
      "auths": {
        "(@= data.values.tap_registry.server @)": {
          "username": "(@= data.values.tap_registry.username @)",
          "password": "(@= data.values.tap_registry.password @)"
        }
      }
    }
EOF
```

Definition of SecretExport to allow the above Secret to be exported to other Namespaces

```yaml
cat <<EOF > tap-install-gitops/kind-iterate/config/tap-repository/secret-export.yaml
apiVersion: secretgen.carvel.dev/v1alpha1
kind: SecretExport
metadata:
  name: tap-registry
  namespace: tap-install
spec:
  toNamespaces:
  - '*'
EOF
```

PackageRepository definition (for TAP 1.3.3)

```yaml
cat <<EOF > tap-install-gitops/kind-iterate/config/tap-repository/pkgr.yaml
#@ load("@ytt:data", "data")
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageRepository
metadata:
  name: tanzu-tap-repository-1.3.3
  namespace: tap-install
spec:
  fetch:
    imgpkgBundle:
      image: #@ "{}/tanzu-application-platform/tap-packages:1.3.3".format(data.values.tap_registry.server)
EOF
```

Definition of App CR for managing resources prepared so far with GitOps. 
Set it to be installed after secretgen-controller. It will be reconciled every 6 hours.


```yaml
cat <<EOF > tap-install-gitops/kind-iterate/kapp/tap-repository.yaml
apiVersion: kappctrl.k14s.io/v1alpha1
kind: App
metadata:
  name: tap-repository
  namespace: kapp
  annotations:
    kapp.k14s.io/change-group: "{name}"
    kapp.k14s.io/change-rule.create-order.0: "upsert after upserting secretgen-controller"
    kapp.k14s.io/change-rule.delete-order.0: "delete before deleting secretgen-controller"
spec:
  syncPeriod: 6h
  serviceAccountName: kapp
  fetch:
  - git:
      url: https://github.com/making/tap-install-gitops.git
      ref: origin/main
      subPath: kind-iterate/config
  template:
  - ytt:
      paths:
      - tap-repository
      valuesFrom:
      - secretRef:
          name: tap-install-gitops
  deploy:
  - kapp:
      rawOptions:
      - --wait-timeout=5m
      - --diff-changes=false
      - --diff-mask=true
EOF
```

The file hierarchy will look like bellow:

```
$ tree tap-install-gitops 
tap-install-gitops
`-- kind-iterate
    |-- config
    |   `-- tap-repository
    |       |-- namespace.yaml
    |       |-- pkgr.yaml
    |       |-- secret-export.yaml
    |       `-- secret.yaml
    `-- kapp
        |-- secretgen-controller.yaml
        `-- tap-repository.yaml
```

#### App CR for TAP's PackageInstall

Define TAP's PackageInstall and Secrets that store tap-values.yaml and overlays in App CRs.

Create a CA certificate to issue self-signed TLS certificates.

```
mkdir -p certs
rm -f certs/*
docker run --rm -v ${PWD}/certs:/certs hitch openssl req -new -nodes -out /certs/ca.csr -keyout /certs/ca.key -subj "/CN=default-ca/O=TAP/C=JP"
chmod og-rwx ca.key
docker run --rm -v ${PWD}/certs:/certs hitch openssl x509 -req -in /certs/ca.csr -days 3650 -extfile /etc/ssl/openssl.cnf -extensions v3_ca -signkey /certs/ca.key -out /certs/ca.crt
```


```
mkdir -p tap-install-gitops/kind-iterate/config/tap/overlays
```

Here the base domain name is 127-0-0-1.sslip.io for Kind. This domain and subdomains are all resolved to 127.0.0.1.


```
export BASE_DOMAIN=127-0-0-1.sslip.io
```

First, define the overlay secret used for TAP installation.


* Overlay for issuing the default TLS certificate

```yaml
cat <<EOF > tap-install-gitops/kind-iterate/config/tap/overlays/contour-default-tls.yaml
#@ load("@ytt:data", "data")
apiVersion: v1
kind: Secret
metadata:
  name: contour-default-tls
  namespace: tap-install
  annotations:
    kapp.k14s.io/change-group: "tap-overlays"
type: Opaque
stringData:
  #@yaml/text-templated-strings
  contour-default-tls.yaml: |
    #@ load("@ytt:data", "data")
    #@ load("@ytt:overlay", "overlay")
    #@ namespace = data.values.namespace
    ---
    apiVersion: v1
    kind: Secret
    metadata:
      name: default-ca
      namespace: #@ namespace
    type: kubernetes.io/tls
    stringData:
      tls.crt: "(@= data.values.default_ca.crt.replace('\n', '\\n') @)"
      tls.key: "(@= data.values.default_ca.key.replace('\n', '\\n') @)"
    ---
    apiVersion: cert-manager.io/v1
    kind: Issuer
    metadata:
      name: default-ca-issuer
      namespace: #@ namespace
    spec:
      ca:
        secretName: default-ca
    ---
    apiVersion: cert-manager.io/v1
    kind: Certificate
    metadata:
      name: tap-default-tls
      namespace: #@ namespace
    spec:
      dnsNames:
      - #@ "*.${BASE_DOMAIN}"
      issuerRef:
        kind: Issuer
        name: default-ca-issuer
      secretName: tap-default-tls
    ---
    apiVersion: projectcontour.io/v1
    kind: TLSCertificateDelegation
    metadata:
      name: contour-delegation
      namespace: #@ namespace
    spec:
      delegations:
      - secretName: tap-default-tls
        targetNamespaces:
        - "*"
EOF
```

* Overlay for default HTTPS in Knative

```yaml
cat <<EOF > tap-install-gitops/kind-iterate/config/tap/overlays/cnrs-https.yaml
apiVersion: v1
kind: Secret
metadata:
  name: cnrs-https
  namespace: tap-install
  annotations:
    kapp.k14s.io/change-group: "tap-overlays"
type: Opaque
stringData:
  cnrs-https.yaml: |
    #@ load("@ytt:overlay", "overlay")
    #@overlay/match by=overlay.subset({"metadata":{"name":"config-network"}, "kind": "ConfigMap"})
    ---
    data:
      #@overlay/match missing_ok=True
      default-external-scheme: https
      #@overlay/match missing_ok=True
      http-protocol: redirected      
EOF
```
* Overlay for deleting Telemetry Pods (:P)

```yaml
cat <<EOF > tap-install-gitops/kind-iterate/config/tap/overlays/tap-telemetry-remove.yaml
apiVersion: v1
kind: Secret
metadata:
  name: tap-telemetry-remove
  namespace: tap-install
  annotations:
    kapp.k14s.io/change-group: "tap-overlays"
type: Opaque
stringData:
  tap-telemetry-remove.yaml: |
    #@ load("@ytt:overlay", "overlay")
    #@overlay/match by=overlay.subset({"metadata":{"namespace":"tap-telemetry"}}), expects="1+"
    #@overlay/remove
    ---     
EOF
```

Then, define the resources that are equivalent to `tanzu package install tap -n tap-install -f tap-values.yaml`.

> Please refer to [this article](https://ik.am/entries/681) for the difference between installing a Package with tanzu CLI and installing a Package by creating a PackageInstall.


Secret definition to store tap-values.yaml

```yaml
cat <<EOF > tap-install-gitops/kind-iterate/config/tap/values.yaml
#@ load("@ytt:data", "data")
#@ load("@ytt:yaml", "yaml")

#@ def tap_values():
shared:
  ingress_domain: ${BASE_DOMAIN}
  image_registry:
    project_path: #@ data.values.image_registry.project_path
    username: #@ data.values.image_registry.username
    password: #@ data.values.image_registry.password
  ca_cert_data: #@ data.values.default_ca.crt

ceip_policy_disclosed: true
profile: iterate

supply_chain: basic

contour:
  contour:
    replicas: 1
  envoy:
    service:
      type: NodePort
      nodePorts:
        http: 31080
        https: 31443
    hostPorts:
      enable: true

cnrs:
  domain_template: "{{.Name}}-{{.Namespace}}.{{.Domain}}"
  default_tls_secret: tanzu-system-ingress/tap-default-tls
  provider: local

package_overlays:
- name: contour
  secrets:
  - name: contour-default-tls
- name: cnrs
  secrets:
  - name: cnrs-https
- name: tap-telemetry
  secrets:
  - name: tap-telemetry-remove

excluded_packages:
- policy.apps.tanzu.vmware.com
- image-policy-webhook.signing.apps.tanzu.vmware.com
- eventing.tanzu.vmware.com
- sso.apps.tanzu.vmware.com
#@ end

apiVersion: v1
kind: Secret
metadata:
  name: tap-tap-install-values
  namespace: tap-install
  annotations:
    kapp.k14s.io/change-group: "tap-install-values"
type: Opaque
stringData:
  tap-values.yaml: #@ yaml.encode(tap_values())
EOF
```

Defining RBAC for installing packages


```yaml
cat <<EOF > tap-install-gitops/kind-iterate/config/tap/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    tkg.tanzu.vmware.com/tanzu-package: tap-tap-install
    kapp.k14s.io/change-group: "tap-rbac"
  name: tap-tap-install-cluster-role
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    tkg.tanzu.vmware.com/tanzu-package: tap-tap-install
    kapp.k14s.io/change-group: "tap-rbac"
  name: tap-tap-install-cluster-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tap-tap-install-cluster-role
subjects:
- kind: ServiceAccount
  name: tap-tap-install-sa
  namespace: tap-install
---
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    tkg.tanzu.vmware.com/tanzu-package: tap-tap-install
    kapp.k14s.io/change-group: "tap-rbac"
  name: tap-tap-install-sa
  namespace: tap-install 
EOF
```

Definition of PackageInstall


```yaml
cat <<EOF > tap-install-gitops/kind-iterate/config/tap/pkgi.yaml
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
  name: tap
  namespace: tap-install
  annotations:
    tkg.tanzu.vmware.com/tanzu-package-ClusterRole: tap-tap-install-cluster-role
    tkg.tanzu.vmware.com/tanzu-package-ClusterRoleBinding: tap-tap-install-cluster-rolebinding
    tkg.tanzu.vmware.com/tanzu-package-Secret: tap-tap-install-values
    tkg.tanzu.vmware.com/tanzu-package-ServiceAccount: tap-tap-install-sa
    kapp.k14s.io/change-group: "{name}"
    kapp.k14s.io/change-rule.create-order.1: "upsert after upserting tap-rbac"
    kapp.k14s.io/change-rule.delete-order.1: "delete before deleting tap-rbac"
    kapp.k14s.io/change-rule.create-order.2: "upsert after upserting tap-values"
    kapp.k14s.io/change-rule.delete-order.2: "delete before deleting tap-values"
    kapp.k14s.io/change-rule.create-order.3: "upsert after upserting tap-overlays"
    kapp.k14s.io/change-rule.delete-order.3: "delete before deleting tap-overlays"
spec:
  syncPeriod: 3h
  serviceAccountName: tap-tap-install-sa
  packageRef:
    refName: tap.tanzu.vmware.com
    versionSelection:
      constraints: 1.3.3
      prereleases: { }
  values:
  - secretRef:
      name: tap-tap-install-values 
EOF
```

Definition of App CR for managing resources prepared so far with GitOps.
 Set it to be installed after PackageRepository. It will be reconciled every hour.


```yaml
cat <<EOF > tap-install-gitops/kind-iterate/kapp/tap.yaml
apiVersion: kappctrl.k14s.io/v1alpha1
kind: App
metadata:
  name: tap
  namespace: kapp
  annotations:
    kapp.k14s.io/change-group: "{name}"
    kapp.k14s.io/change-rule.create-order.0: "upsert after upserting tap-repository"
    kapp.k14s.io/change-rule.delete-order.0: "delete before deleting tap-repository"
spec:
  syncPeriod: 1h
  serviceAccountName: kapp
  fetch:
  - git:
      url: https://github.com/making/tap-install-gitops.git
      ref: origin/main
      subPath: kind-iterate/config
  template:
  - ytt:
      paths:
      - tap
      valuesFrom:
      - secretRef:
          name: tap-install-gitops
  deploy:
  - kapp:
      rawOptions:
      - --wait-timeout=5m
      - --diff-changes=false
      - --diff-mask=true
EOF
```

The file hierarchy will look like bellow:

```
$ tree tap-install-gitops 
tap-install-gitops
`-- kind-iterate
    |-- config
    |   |-- tap
    |   |   |-- overlays
    |   |   |   |-- cnrs-https.yaml
    |   |   |   |-- contour-default-tls.yaml
    |   |   |   `-- tap-telemetry-remove.yaml
    |   |   |-- pkgi.yaml
    |   |   |-- rbac.yaml
    |   |   `-- values.yaml
    |   `-- tap-repository
    |       |-- namespace.yaml
    |       |-- pkgr.yaml
    |       |-- secret-export.yaml
    |       `-- secret.yaml
    `-- kapp
        |-- secretgen-controller.yaml
        |-- tap-repository.yaml
        `-- tap.yaml
```

#### A parent App CR that collectively manages a group of App CRs

App CR definition for managing the App CR group defined so far with GitOps. 
It will be reconciled every 10 minutes.


> ☝️ It follows [app of apps pattern](https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/)

```yaml
cat <<EOF > tap-install-gitops/kind-iterate/apps.yaml
apiVersion: kappctrl.k14s.io/v1alpha1
kind: App
metadata:
  name: apps
  namespace: kapp
spec:
  serviceAccountName: kapp
  fetch:
  - git:
      url: https://github.com/making/tap-install-gitops.git
      ref: origin/main
      subPath: kind-iterate/kapp
  syncPeriod: 10m
  template:
  - ytt:
      paths:
      - '.'
  deploy:
  - kapp:
      rawOptions:
      - --wait-timeout=5m
      - --diff-changes=true
      - --diff-mask=false
EOF
```

The file hierarchy will look like bellow:


```
$ tree tap-install-gitops                              
tap-install-gitops
`-- kind-iterate
    |-- apps.yaml
    |-- config
    |   |-- tap
    |   |   |-- overlays
    |   |   |   |-- cnrs-https.yaml
    |   |   |   |-- contour-default-tls.yaml
    |   |   |   `-- tap-telemetry-remove.yaml
    |   |   |-- pkgi.yaml
    |   |   |-- rbac.yaml
    |   |   `-- values.yaml
    |   `-- tap-repository
    |       |-- namespace.yaml
    |       |-- pkgr.yaml
    |       |-- secret-export.yaml
    |       `-- secret.yaml
    `-- kapp
        |-- secretgen-controller.yaml
        |-- tap-repository.yaml
        `-- tap.yaml
```

### Push manifests to a git repository


Push the created manifest to https://github.com/making/tap-install-gitops .

```
GITHUB_USERNAME=...

cd tap-install-gitops 
git init
git add -A
git commit -m "first commit"
git branch -M main
git remote add origin git@github.com:${GITHUB_USERNAME}/tap-install-gitops.git
git push -u origin main
cd ..
```

### Creatr a Secret for credentials

Git-managed manifests shoule not include credentials, but reference externalized parameters.
Externalized parameters are set to be referenced from Secret. This Secret is outside Git management and is created as follows:

```yaml
TANZUNET_USERNAME=...
TANZUNET_PASSWORD=...
GITHUB_USERNAME=...
GITHUB_API_TOKEN=...

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: tap-install-gitops
  namespace: kapp
type: Opaque
stringData:
  credentials.yaml: |
    tap_registry:
      server: registry.tanzu.vmware.com
      username: ${TANZUNET_USERNAME}
      password: ${TANZUNET_PASSWORD}
    image_registry:
      project_path: ghcr.io/${GITHUB_USERNAME}
      username: ${GITHUB_USERNAME}
      password: ${GITHUB_API_TOKEN}
    default_ca:
      crt: |
$(cat certs/ca.crt | sed 's/^/        /g')
      key: |
$(cat certs/ca.key | sed 's/^/        /g')
EOF
```

You can also include "encrypted" credentials in the git repo instead of using Secret out of the git repository.
kapp-controller supports [sops](https://github.com/mozilla/sops). See https://carvel.dev/kapp-controller/docs/v0.43.2/sops/ for how to do this.

> This blog is managed using GitOps using sops.<br>
> The file structure is slightly different from this article, but https://github.com/categolj/k8s-manifests will be helpful

### Install TAP

Now that we have a Git repository and externalized parameters, it's time to install TAP with App CR.
Just create the parent App CR and everything will be installed.


```
kubectl apply -f tap-install-gitops/kind-iterate/apps.yaml
```

Check the progress until it completes with the following command:

```
while [ "$(kubectl -n kapp get app apps -o=jsonpath='{.status.friendlyDescription}')" != "Reconcile succeeded" ];do
  date
  kubectl get app -A
  echo "---------------------------------------------------------------------"
  sleep 10
done
echo "✅ Install succeeded"
```

After a successful install, each app will reconcile at syncPeriod intervals. If you can't wait for the syncPeriod, use the bellow command to trigger the reconciliation forcely:

```
kctrl app kick -n kapp -a <App name>
```
You can download the `kctrl` CLI from the [kapp-controller Releases page](https://github.com/vmware-tanzu/carvel-kapp-controller/releases).

### Uninstall TAP

Simply deleting the parent App CR will uninstall everything.


```
kubectl delete -f tap-install-gitops/kind-iterate/apps.yaml
```

---

I introduced how to GitOps TAP with kapp-controller.


Using this method, to install TAP, you just need to


* Install kapp-controller
* Create Service Account
* Register Secret of credentials (or Register [GPG Private Key](https://carvel.dev/kapp-controller/docs/v0.43.2/sops/#using-gpg-2) when using sops)
* Create an apps App CR


No operation by tanzu CLI is required.


Software other than TAP (Prometheus, etc.) can be installed at the same time, making it easier to automate environment construction. 
It will also be useful if you want to repeatedly build and destroy the TAP testing environment.