For Tanzu Application Platform (as of 1.6), as documentation here stated, Out of the Box (OOTB) Supply Chains supports the following Workload Types:
type=web... intended for scalable web applications. A Knative Service resource is created. Scale to Zero, Zero to N are supported.type=server... intended for traditional web applications. K8s standard Deployment and Service resources are created.type=worker... intended for background applications processing queues. A K8s standard Deployment is created.
If you deploy your app with type=server, you need to create an Ingress resource somehow if you want to expose your app in Ingress.
We will introduce three methods below.
- Create an Ingress resource directly with
kubectl - Add Ingress to resources created in OOTB SupplyChain
- Use Carvel Package Supply Chain
table of contents
- Create an Ingress resource directly with
kubectl - Add Ingress to resources created in OOTB SupplyChain
- Using the Carvel Package Supply Chain
Create an Ingress resource directly with kubectl
First, create a Workload for the sample application.
tanzu apps workload apply hello-nodejs \
--app hello-nodejs \
--git-repo https://github.com/making/hello-nodejs \
--git-branch master \
--type server \
-n demo
You can see the manifest created from this workload with the following command:
kubectl get cm -n demo hello-nodejs-server -ojsonpath='{.data.delivery\.yml}'
You will get the following output:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-nodejs
annotations:
kapp.k14s.io/update-strategy: fallback-on-replace
ootb.apps.tanzu.vmware.com/servicebinding-workload: "true"
kapp.k14s.io/change-rule: upsert after upserting servicebinding.io/ServiceBindings
labels:
app.kubernetes.io/part-of: hello-nodejs
apps.tanzu.vmware.com/workload-type: server
app.kubernetes.io/component: run
carto.run/workload-name: hello-nodejs
spec:
selector:
matchLabels:
app.kubernetes.io/component: run
app.kubernetes.io/part-of: hello-nodejs
apps.tanzu.vmware.com/workload-type: server
carto.run/workload-name: hello-nodejs
template:
metadata:
annotations:
conventions.carto.run/applied-conventions: |-
appliveview-sample/app-live-view-appflavour-check
spring-boot-convention/auto-configure-actuators-check
spring-boot-convention/app-live-view-appflavour-check
developer.conventions/target-containers: workload
labels:
app.kubernetes.io/component: run
app.kubernetes.io/part-of: hello-nodejs
apps.tanzu.vmware.com/workload-type: server
carto.run/workload-name: hello-nodejs
spec:
containers:
- image: ghcr.io/making/workloads/hello-nodejs-demo@sha256:052ffee7966eeda9cf3a0d6a255f70b443ed0c39ab24591ab4b9ab857b30995b
name: workload
resources: {}
securityContext:
runAsUser: 1000
serviceAccountName: default
---
apiVersion: v1
kind: Service
metadata:
name: hello-nodejs
labels:
app.kubernetes.io/part-of: hello-nodejs
apps.tanzu.vmware.com/workload-type: server
app.kubernetes.io/component: run
carto.run/workload-name: hello-nodejs
spec:
selector:
app.kubernetes.io/component: run
app.kubernetes.io/part-of: hello-nodejs
apps.tanzu.vmware.com/workload-type: server
carto.run/workload-name: hello-nodejs
ports:
- targetPort: 8080
port: 8080
name: http
For example, if you want to expose this app on hello-nodejs-demo.tap.192-168-228-200.sslip.io (assuming that envoy's ExternalIP is 192.168.228.201), you can create an ingress resource with the following command. In the case of TAP multi cluster topology, please execute in Run cluster.
Note that app-editor (which is usually assigned to an app developer) does not have permission to execute this command. app-operator is allowed to do that instead.
- When not to use TLS
kubectl create ingress hello-nodejs -n demo \
--rule="hello-nodejs-demo.tap.192-168-228-200.sslip.io/*=hello-nodejs:8080"
Access the app.
$ curl -sv http://hello-nodejs-demo.tap.192-168-228-200.sslip.io
> GET / HTTP/1.1
> Host: hello-nodejs-demo.tap.192-168-228-200.sslip.io
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/1.1 200 OK
< x-powered-by: Express
< content-type: text/html; charset=utf-8
< content-length: 14
< etag: W/"e-LQS9yOYOT+WGITCx9XjB8GC9nDI"
< date: Thu, 10 Aug 2023 06:03:01 GMT
< x-envoy-upstream-service-time: 5
< server: envoy
<
Hello World!!
- When issuing a TLS certificate with cert-manager
kubectl create ingress hello-nodejs -n demo \
--rule="hello-nodejs-demo.tap.192-168-228-200.sslip.io/*=hello-nodejs:8080,tls=hello-nodejs-tls" \
--annotation cert-manager.io/cluster-issuer=tap-ingress-selfsigned
The above annotation assumes that there is a ClusterIssuer named tap-ingress-selfsigned.
Access the app.
$ curl -skv https://hello-nodejs-demo.tap.192-168-228-200.sslip.io
> GET / HTTP/2
> Host: hello-nodejs-demo.tap.192-168-228-200.sslip.io
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/2 200
< x-powered-by: Express
< content-type: text/html; charset=utf-8
< content-length: 14
< etag: W/"e-LQS9yOYOT+WGITCx9XjB8GC9nDI"
< date: Thu, 10 Aug 2023 06:03:53 GMT
< x-envoy-upstream-service-time: 5
< server: envoy
<
Hello World!!
- When using Contour's TLS Certificate Delegation
kubectl create ingress hello-nodejs -n demo \
--rule="hello-nodejs-demo.tap.192-168-228-200.sslip.io/*=hello-nodejs:8080,tls=tap-default-tls" \
--annotation projectcontour.io/tls-cert-namespace=tanzu-system-ingress
With the above annotation, it is assumed that tanzu-system-ingress namespace has a Secret called tap-default-tls containing a Wildcard certificate for *.tap.192-168-228-200.sslip.io, and it can be referred from any namespace via TLSCertificateDelegation.
If you want to manage YAML instead of running kubectl create, just add the --dry-run=client -oyaml options to the above command.
Also, if you want to use an Ingress other than Contour, specify the IngressClass name with --class.
For example, if ingress-nginx is installed in TAP's Run cluster, you can see the IngressClass name as below:
$ kubectl get ingressclass
NAME CONTROLLER PARAMETERS AGE
nginx k8s.io/ingress-nginx <none> 3m30s
Add --class=nginx as follows. (It is assumed that the ExternalIP of ingress-nginx-controller is 192.168.228.201.)
kubectl create ingress hello-nodejs -n demo \
--rule="hello-nodejs-demo.tap.192-168-228-201.sslip.io/*=hello-nodejs:8080" \
--class=nginx
Access the app.
$ curl -sv http://hello-nodejs-demo.tap.192-168-228-201.sslip.io
> GET / HTTP/1.1
> Host: hello-nodejs-demo.tap.192-168-228-201.sslip.io
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 10 Aug 2023 06:11:16 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 14
< Connection: keep-alive
< X-Powered-By: Express
< ETag: W/"e-LQS9yOYOT+WGITCx9XjB8GC9nDI"
<
Hello World!!
This method does not require TAP customization and is pretty straightforward, but by default the developer (app-editor) cannot create an Ingress, so ask the app-operator to create it, or you will need to add the following permissions to app-editor.
kubectl apply -f - << 'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: app-editor-with-ingress
labels:
apps.tanzu.vmware.com/aggregate-to-app-editor: "true"
rules:
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- create
- patch
- update
- delete
- deletecollection
EOF
In addition, you will have to manage the Ingress resource separately.
These issues can be solved by adding Ingress to the resources created in the OOTB SupplyChain described below.
Add Ingress to resources created in OOTB SupplyChain
As described in the document below, you can add Ingress to the manifest generated from Workload by customizing the ClusterConfigTemplate.
In this case, you have the option of adding a new Workload Type like type=server-ingress, or adding an Ingress to an existing type=server.
Add a new Workload Type
Let's create a new workload type server-ingress. Create server-ingress-template by copying an existing server-template.
Define the server-ingress-template below. The difference from server-template is shown by <------------ Added ---------------->.
apiVersion: carto.run/v1alpha1
kind: ClusterConfigTemplate
metadata:
name: server-ingress-template
spec:
configPath: .data
params:
- name: ports
default:
- containerPort: 8080
port: 8080
name: http
healthRule:
alwaysHealthy: {}
ytt: |
#@ load("@ytt:data", "data")
#@ load("@ytt:yaml", "yaml")
#@ load("@ytt:struct", "struct")
#@ load("@ytt:assert", "assert")
#@ def merge_labels(fixed_values):
#@ labels = {}
#@ if hasattr(data.values.workload.metadata, "labels"):
#@ labels.update(data.values.workload.metadata.labels)
#@ end
#@ labels.update(fixed_values)
#@ return labels
#@ end
#@ def intOrString(v):
#@ return v if type(v) == "int" else int(v.strip()) if v.strip().isdigit() else v
#@ end
#@ def merge_ports(ports_spec, containers):
#@ ports = {}
#@ for c in containers:
#@ for p in getattr(c, "ports", []):
#@ ports[p.containerPort] = {"targetPort": p.containerPort, "port": p.containerPort, "name": getattr(p, "name", str(p.containerPort))}
#@ end
#@ end
#@ for p in ports_spec:
#@ targetPort = getattr(p, "containerPort", p.port)
#@ type(targetPort) in ("string", "int") or fail("containerPort must be a string or int")
#@ targetPort = intOrString(targetPort)
#@
#@ port = p.port
#@ type(port) in ("string", "int") or fail("port must be a string or int")
#@ port = int(port)
#@ ports[p.port] = {"targetPort": targetPort, "port": port, "name": getattr(p, "name", str(p.port))}
#@ end
#@ return ports.values()
#@ end
#! <------------ Added ---------------->
#@ def merge_annotations(fixed_values):
#@ annotations = {}
#@ if hasattr(data.values.params, "annotations"):
#@ annotations.update(data.values.params.annotations)
#@ end
#@ annotations.update(fixed_values)
#@ return annotations
#@ end
#! <------------ Added ---------------->
#@ def delivery():
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: #@ data.values.workload.metadata.name
annotations:
kapp.k14s.io/update-strategy: "fallback-on-replace"
ootb.apps.tanzu.vmware.com/servicebinding-workload: "true"
kapp.k14s.io/change-rule: "upsert after upserting servicebinding.io/ServiceBindings"
labels: #@ merge_labels({ "app.kubernetes.io/component": "run", "carto.run/workload-name": data.values.workload.metadata.name })
spec:
selector:
matchLabels: #@ data.values.config.metadata.labels
template: #@ data.values.config
---
apiVersion: v1
kind: Service
metadata:
name: #@ data.values.workload.metadata.name
labels: #@ merge_labels({ "app.kubernetes.io/component": "run", "carto.run/workload-name": data.values.workload.metadata.name })
spec:
selector: #@ data.values.config.metadata.labels
ports:
#@ hasattr(data.values.params, "ports") and len(data.values.params.ports) or assert.fail("one or more ports param must be provided.")
#@ declared_ports = {}
#@ if "ports" in data.values.params:
#@ declared_ports = data.values.params.ports
#@ else:
#@ declared_ports = struct.encode([{ "containerPort": 8080, "port": 8080, "name": "http"}])
#@ end
#@ for p in merge_ports(declared_ports, data.values.config.spec.containers):
- #@ p
#@ end
#! <------------ Added ---------------->
#@ ingress_domain = "tap.192-168-228-200.sslip.io"
#@ cluster_issuer = "tap-ingress-selfsigned"
#@ port = intOrString(data.values.params.ports[0].port) if hasattr(data.values.params, "ports") and len(data.values.params.ports) > 0 and hasattr(data.values.params.ports[0], "port") else 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: #@ data.values.workload.metadata.name
annotations: #@ merge_annotations({"cert-manager.io/cluster-issuer": cluster_issuer, "kapp.k14s.io/change-rule": "upsert after upserting Services"})
labels: #@ merge_labels({ "app.kubernetes.io/component": "run", "carto.run/workload-name": data.values.workload.metadata.name })
spec:
tls:
- secretName: #@ "{}-tls".format(data.values.workload.metadata.name)
hosts:
- #@ "{}-{}.{}".format(data.values.workload.metadata.name, data.values.workload.metadata.namespace, ingress_domain)
rules:
- host: #@ "{}-{}.{}".format(data.values.workload.metadata.name, data.values.workload.metadata.namespace, ingress_domain)
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: #@ data.values.workload.metadata.name
port:
number: #@ port
#! <------------ Added ---------------->
#@ end
---
apiVersion: v1
kind: ConfigMap
metadata:
name: #@ data.values.workload.metadata.name + "-server"
labels: #@ merge_labels({ "app.kubernetes.io/component": "config" })
data:
delivery.yml: #@ yaml.encode(delivery())
Save this content to server-ingress-template.yaml and apply. In the case of TAP multi-cluster topology, execute it in the Build cluster.
kubectl apply -f server-ingress-template.yaml
Also add a ClusterRole with the following command so that Deliverable can create Ingress resources generated from the SupplyChain. In the case of TAP multi cluster topology, please run the command in Run clusters.
cat << 'EOF' > deliverable-with-ingress.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: deliverable-with-ingress
labels:
apps.tanzu.vmware.com/aggregate-to-deliverable: "true"
rules:
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- create
- patch
- update
- delete
- deletecollection
EOF
kubectl apply -f deliverable-with-ingress.yaml
Add the following contents to tap-values.yaml.
ootb_supply_chain_basic:
supported_workloads:
- type: web
cluster_config_template_name: config-template
- type: server
cluster_config_template_name: server-template
- type: worker
cluster_config_template_name: worker-template
- type: server-ingress #! <--- Added
cluster_config_template_name: server-ingress-template
The key to set is ootb_supply_chain_<suppy_chain> and the example above is for supply_chain: basic. So the key here is ootb_supply_chain_basic.
Update TAP with the following command: In the case of TAP multi-cluster topology, execute it in the Build cluster.
tanzu package installed update -n tap-install tap --values-file tap-values.yaml
Check that server-ingress has been added to the available Workload Types by running the following command:
$ tanzu apps cluster-supply-chain get source-to-url
---
# source-to-url: Ready
---
Supply Chain Selectors
TYPE KEY OPERATOR VALUE
expressions apps.tanzu.vmware.com/workload-type In web
expressions apps.tanzu.vmware.com/workload-type In server
expressions apps.tanzu.vmware.com/workload-type In worker
expressions apps.tanzu.vmware.com/workload-type In server-ingress
expressions apps.tanzu.vmware.com/carvel-package-workflow DoesNotExist
Create a workload of type=server-ingress with the following command.
kubectl delete ingress hello-nodejs -n demo # if exists
tanzu apps workload apply hello-nodejs \
--app hello-nodejs \
--git-repo https://github.com/making/hello-nodejs \
--git-branch master \
--type server-ingress \
-n demo
You can check the YAML created by Workload with the following command:
kubectl get cm -n demo hello-nodejs-server -ojsonpath='{.data.delivery\.yml}'
It will output YAML like this: You'll see that Ingress has been added.
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-nodejs
annotations:
kapp.k14s.io/update-strategy: fallback-on-replace
ootb.apps.tanzu.vmware.com/servicebinding-workload: "true"
kapp.k14s.io/change-rule: upsert after upserting servicebinding.io/ServiceBindings
labels:
app.kubernetes.io/part-of: hello-nodejs
apps.tanzu.vmware.com/workload-type: server-ingress
app.kubernetes.io/component: run
carto.run/workload-name: hello-nodejs
spec:
selector:
matchLabels:
app.kubernetes.io/component: run
app.kubernetes.io/part-of: hello-nodejs
apps.tanzu.vmware.com/workload-type: server-ingress
carto.run/workload-name: hello-nodejs
template:
metadata:
annotations:
conventions.carto.run/applied-conventions: |-
appliveview-sample/app-live-view-appflavour-check
spring-boot-convention/auto-configure-actuators-check
spring-boot-convention/app-live-view-appflavour-check
developer.conventions/target-containers: workload
labels:
app.kubernetes.io/component: run
app.kubernetes.io/part-of: hello-nodejs
apps.tanzu.vmware.com/workload-type: server-ingress
carto.run/workload-name: hello-nodejs
spec:
containers:
- image: ghcr.io/making/workloads/hello-nodejs-demo@sha256:052ffee7966eeda9cf3a0d6a255f70b443ed0c39ab24591ab4b9ab857b30995b
name: workload
resources: {}
securityContext:
runAsUser: 1000
serviceAccountName: default
---
apiVersion: v1
kind: Service
metadata:
name: hello-nodejs
labels:
app.kubernetes.io/part-of: hello-nodejs
apps.tanzu.vmware.com/workload-type: server-ingress
app.kubernetes.io/component: run
carto.run/workload-name: hello-nodejs
spec:
selector:
app.kubernetes.io/component: run
app.kubernetes.io/part-of: hello-nodejs
apps.tanzu.vmware.com/workload-type: server-ingress
carto.run/workload-name: hello-nodejs
ports:
- targetPort: 8080
port: 8080
name: http
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hello-nodejs
annotations:
cert-manager.io/cluster-issuer: tap-ingress-selfsigned
kapp.k14s.io/change-rule: upsert after upserting Services
labels:
app.kubernetes.io/part-of: hello-nodejs
apps.tanzu.vmware.com/workload-type: server-ingress
app.kubernetes.io/component: run
carto.run/workload-name: hello-nodejs
spec:
tls:
- secretName: hello-nodejs-tls
hosts:
- hello-nodejs-demo.tap.192-168-228-200.sslip.io
rules:
- host: hello-nodejs-demo.tap.192-168-228-200.sslip.io
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: hello-nodejs
port:
number: 8080
You can confirm that the Ingress resource has been created from the supply chain.
$ kubectl get ing -n demo --show-labels
NAME CLASS HOSTS ADDRESS PORTS AGE LABELS
hello-nodejs <none> hello-nodejs-demo.tap.192-168-228-200.sslip.io 192.168.228.200 80, 443 96s app.kubernetes.io/component=run,app.kubernetes.io/part-of=hello-nodejs,apps.tanzu.vmware.com/workload-type=server-ingress,carto.run/workload-name=hello-nodejs,kapp.k14s.io/app=1691647255322645686,kapp.k14s.io/association=v1.4aa1edeb0a707e6a5e81cfde4668a7b0
Access the app.
$ curl -ks https://hello-nodejs-demo.tap.192-168-228-200.sslip.io
Hello World!!
If you want to add ClusterConfigTemplate and ClusterRole together at the timing of tanzu package install, you can register the file as an overlay in Secret as follows.
# In the Build Cluster in case of the Muti Cluster toplogy
kubectl -n tap-install create secret generic ootb-templates-server-ingress-template \
-o yaml \
--dry-run=client \
--from-file=server-ingress-template.yaml \
| kubectl apply -f-
# In Run Clusters in case of the Muti Cluster toplogy
kubectl -n tap-install create secret generic tap-auth-deliverable-with-ingress \
-o yaml \
--dry-run=client \
--from-file=deliverable-with-ingress.yaml \
| kubectl apply -f-
Add the following settings to tap-values.yaml.
package_overlays:
- name: ootb-templates
secrets:
- name: ootb-templates-server-ingress-template # In the Build Cluster in case of the Muti Cluster toplogy
- name: tap-auth
secrets:
- name: tap-auth-deliverable-with-ingress # In Run Clusters in case of the Muti Cluster toplogy
Update TAP.
tanzu package installed update -n tap-install tap --values-file tap-values.yaml
Add an Ingress to an existing type=server
The method of adding a new Workload Type allows you to manage the template yourself, so it is easy to customize,
But if the copied type=server template is updated, you need to reflect the updates. This problem can be solved by modifying the existing workload type so that it can also create an ingress instead of adding a new workload type.
Modify the existing ClusterConfigTemplate server-template by applying the following overlay.
apiVersion: v1
kind: Secret
metadata:
name: ootb-templates-overlay-ingress
namespace: tap-install
type: Opaque
stringData:
overlay-ingress.yaml: |
#@ def merge_annotations_def_string():
#@ return '''
#@ #@ def merge_annotations(fixed_values):
#@ #@ annotations = {}
#@ #@ if hasattr(data.values.params, "annotations"):
#@ #@ annotations.update(data.values.params.annotations)
#@ #@ end
#@ #@ annotations.update(fixed_values)
#@ #@ return annotations
#@ #@ end
#@ '''
#@ end
#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")
#@ ingress_domain = data.values.ingress_domain
#@ cluster_issuer = data.values.cluster_issuer
#@overlay/match by=overlay.subset({"kind":"ClusterConfigTemplate", "metadata": {"name": "server-template"}})
---
spec:
#@overlay/replace via=lambda left, right: "{}\n{}".format(left.replace("#@ def delivery():", '\n'.join([ merge_annotations_def_string(), "#@ def delivery():"])), '\n'.join([' {}'.format(x) for x in right.replace("INGRESS_DOMAIN", ingress_domain).replace("CLUSTER_ISSUER", cluster_issuer).split('\n')]))
ytt: |
#@yaml/text-templated-strings
ingress.yml: |
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: (@= data.values.workload.metadata.name @)
annotations:
(@= '\n'.join([' {}'.format(x) for x in yaml.encode(merge_annotations({ "cert-manager.io/cluster-issuer": "CLUSTER_ISSUER", "kapp.k14s.io/change-rule": "upsert after upserting Services" })).split('\n')]) @)
labels:
(@= '\n'.join([' {}'.format(x) for x in yaml.encode(merge_labels({ "app.kubernetes.io/component": "run", "carto.run/workload-name": data.values.workload.metadata.name })).split('\n')]) @)
spec:
tls:
- secretName: (@= data.values.workload.metadata.name @)-tls
hosts:
- (@= "{}-{}.{}".format(data.values.workload.metadata.name, data.values.workload.metadata.namespace, "INGRESS_DOMAIN") @)
rules:
- host: (@= "{}-{}.{}".format(data.values.workload.metadata.name, data.values.workload.metadata.namespace, "INGRESS_DOMAIN") @)
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: (@= data.values.workload.metadata.name @)
port:
number: (@= str(data.values.params.ports[0].port) if hasattr(data.values.params, "ports") and len(data.values.params.ports) > 0 and hasattr(data.values.params.ports[0], "port") else "8080" @)
Save this file to ootb-templates-overlay-ingress.yaml and apply.
kubectl apply -f ootb-templates-overlay-ingress.yaml
Add the following settings to tap-values.yaml.
ootb_templates:
ingress_domain: tap.192-168-228-200.sslip.io
cluster_issuer: tap-ingress-selfsigned
package_overlays:
- name: ootb-templates # In Build Cluster in case of the Muti Cluster toplogy
secrets:
- name: ootb-templates-overlay-ingress
Update TAP.
tanzu package installed update -n tap-install tap --values-file tap-values.yaml
Create a Workload with type=server.
tanzu apps workload apply hello-nodejs \
--app hello-nodejs \
--git-repo https://github.com/making/hello-nodejs \
--git-branch master \
--type server \
-n demo
You can check the Ingress YAML created by Workload with the following command:
kubectl get cm -n demo hello-nodejs-server -ojsonpath='{.data.ingress\.yml}'
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hello-nodejs
annotations:
cert-manager.io/cluster-issuer: tap-ingress-selfsigned
kapp.k14s.io/change-rule: upsert after upserting Services
labels:
app.kubernetes.io/part-of: hello-nodejs
apps.tanzu.vmware.com/workload-type: server
app.kubernetes.io/component: run
carto.run/workload-name: hello-nodejs
spec:
tls:
- secretName: hello-nodejs-tls
hosts:
- hello-nodejs-demo.tap.192-168-228-200.sslip.io
rules:
- host: hello-nodejs-demo.tap.192-168-228-200.sslip.io
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: hello-nodejs
port:
number: 8080
You can confirm that the Ingress resource has been created from the supply chain.
$ kubectl get ing -n demo --show-labels
NAME CLASS HOSTS ADDRESS PORTS AGE LABELS
hello-nodejs <none> hello-nodejs-demo.tap.192-168-228-200.sslip.io 192.168.228.200 80, 443 154m app.kubernetes.io/component=run,app.kubernetes.io/part-of=hello-nodejs,apps.tanzu.vmware.com/workload-type=server,carto.run/workload-name=hello-nodejs,kapp.k14s.io/app=1691647255322645686,kapp.k14s.io/association=v1.4aa1edeb0a707e6a5e81cfde4668a7b0
Compared to adding a new Workload Type, the method of updating an existing Workload Type automatically merges changes in the copy source template when TAP is upgraded, so there is no need to reflect the changes.
On the other hand, there is no guarantee if overlay will still work in the next version. It is necessary to verify the operation when upgrading the version.
The method of generating Ingress from Supply Chain is convenient because it automatically creates an Ingress resource, but the limitation is that the host name of the Ingress created at the time of creating a Workload is fixed.
In this example, hello-nodejs-demo.tap.192-168-228-200.sslip.io is hardcoded in the Ingress resource.
If there is only one environment (such as Run cluster) where the app runs, there is no problem, but if you want to deploy to multiple Run clusters, it becomes a problem. Manifest generated from Supply Chain cannot be changed with parameters according to the environment.
As a workaround when using Gitops mode, for another Run cluster, if you create the following overlay on the Git repository, the host name will be overwritten when deploying.
#@ load("@ytt:overlay", "overlay")
#@ ingress_domain = "hello-nodejs-demo.production.example.com"
#@overlay/match by=overlay.subset({"kind":"Ingress"})
---
spec:
tls:
#@overlay/match by=overlay.index(0)
- hosts:
#@overlay/match by=overlay.index(0)
- #@ ingress_domain
rules:
#@overlay/match by=overlay.index(0)
- host: #@ ingress_domain
Using the Carvel Package Supply Chain
[Carvel Package Supply Chains](https://docs.vmware.com/en/VMware-Tanzu-Application -Platform/1.6/tap/scc-config-deploy-multi-env.html) was introduced as an Alpha version in TAP 1.5 and is a Beta version as of TAP 1.6. It is still an experimental feature with many limitations and is only available in the OOTB Basic Supply Chain.
Carvel Package Supply Chains create and push a manifest like a normal Supply Chain, but instead of creating a Deliverable to deploy that manifest, it creates a Carvel Package resource.
Install this Package into an app environment (such as a Run cluster). By passing the parameters for the PackageInstall resource created at this time, you will be able to change the parameters according to the execution environment.
Carvel Package Supply Chains also creates an Ingress resource if type=server.
As of TAP 1.6, manifests can only be pushed to Git Repository, so [Gitops mode](https://docs.vmware.com/en/VMware-Tanzu-Application-Platform/1.6/tap/scc-gitops- vs-regops.html#gitops-0) is required.
To enable Carvel Package Supply Chains, you need to explicitly add the following settings to tap-values.yaml. For multi cluster topology, this setting is required in the Build cluster.
ootb_supply_chain_basic:
carvel_package:
workflow_enabled: true
Enable this setting and update the TAP with the following command:
tanzu package installed update -n tap-install tap --values-file tap-values.yaml
⚠️ Please delete the above overlay setting of
ootb-templates-overlay-ingressfromtap-values.yaml, as Ingress creation will be duplicated.
Check the Supply Chain list and you will see that source-to-url-package has been added.
$ tanzu apps cluster-supply-chain list
NAME READY AGE
basic-image-to-url Ready 6h31m
basic-image-to-url-package Ready 89s
source-to-url Ready 6h31m
source-to-url-package Ready 89s
You can check the parameters for using this Supply Chain, and you can see that you should set apps.tanzu.vmware.com/carvel-package-workflow=true in the label like this:
$ tanzu apps cluster-supply-chain get source-to-url-package
---
# source-to-url-package: Ready
---
Supply Chain Selectors
TYPE KEY OPERATOR VALUE
expressions apps.tanzu.vmware.com/workload-type In web
expressions apps.tanzu.vmware.com/workload-type In server
expressions apps.tanzu.vmware.com/workload-type In worker
expressions apps.tanzu.vmware.com/carvel-package-workflow In true
Then create a Workload using Carvel Package Supply Chain with the following command. The configuration for using GitOps mode is omitted in this article.
tanzu apps workload apply hello-nodejs \
--app hello-nodejs \
--git-repo https://github.com/making/hello-nodejs \
--git-branch master \
--label apps.tanzu.vmware.com/carvel-package-workflow=true \
--type server \
--param gitops_branch=main \
--param gitops_commit_message=Bump \
--param gitops_server_address=https://github.com \
--param gitops_repository_owner=making \
--param gitops_repository_name=tap-gitops-manifests \
--param gitops_user_email=makingx+bot@gmail.com \
--param gitops_user_name=making-bot \
--param gitops_ssh_secret=git-basic \
-n demo
Once the workload is ready, it will create the following resources:
$ tanzu apps workload get hello-nodejs --namespace demo
📡 Overview
name: hello-nodejs
type: server
namespace: demo
💾 Source
type: git
url: https://github.com/making/hello-nodejs
branch: master
revision: master@sha1:fde413c0fba0003c218a60bde69c8e254d3b15a6
📦 Supply Chain
name: source-to-url-package
NAME READY HEALTHY UPDATED RESOURCE
source-provider True True 6m25s gitrepositories.source.toolkit.fluxcd.io/hello-nodejs
image-provider True True 5m29s images.kpack.io/hello-nodejs
config-provider True True 5m22s podintents.conventions.carto.run/hello-nodejs
app-config True True 5m22s configmaps/hello-nodejs-server
service-bindings True True 5m22s configmaps/hello-nodejs-with-claims
api-descriptors True True 5m22s configmaps/hello-nodejs-with-api-descriptors
carvel-package True True 5m8s taskruns.tekton.dev/hello-nodejs-carvel-package-j86xm
config-writer True True 4m55s runnables.carto.run/hello-nodejs-pkg-cfg-writer
🚚 Delivery
Delivery resources not found.
💬 Messages
No messages found.
🛶 Pods
NAME READY STATUS RESTARTS AGE
hello-nodejs-build-1-build-pod 0/1 Completed 0 6m27s
hello-nodejs-carvel-package-j86xm-pod 0/3 Completed 0 5m22s
hello-nodejs-pkg-cfg-writer-jsg9x-pod 0/2 Completed 0 5m6s
hello-nodejs-carvel-package-**** bundles the manifest with imgpkg and pushes it, and hello-nodejs-pkg-cfg-writer-**** converts the imgpkgBundle into a package resource Define it and push it to your git repository.
The actual pushed Package manifest is https://github.com/making/tap-gitops-manifests/blob/main/hello-nodejs.demo.tap/packages/20230811080453.0.0.yml.
You can download the bundled manifest with the command: (The visibility is set to public, so you can actually run it and check the contents.)
imgpkg pull -b ghcr.io/making/workloads/hello-nodejs-demo-bundle@sha256:b0a013325c4c089befffd72643d576fb1109bebadce87f2f7b3615aef5ed9d75 -o /tmp/hello-nodejs
Then apply the Package resource pushed on Git. For multi cluster topology, apply to Run cluster.
The documentation recommends using GitOps tools here,
The former two are available in TAP without any additional installation.
Since we're skipping the setup in this article, instead of using GitOps, simply create the Package resource with kubectl apply as follows.
kubectl apply -f https://github.com/making/tap-gitops-manifests/raw/main/hello-nodejs.demo.tap/packages/20230811080453.0.0.yml -n demo
Now we have the following Package resources:
$ kubectl get package -n demo
NAME PACKAGEMETADATA NAME VERSION AGE
hello-nodejs.demo.tap.20230811080453.0.0+build.fde413c hello-nodejs.demo.tap 20230811080453.0.0+build.fde413c 5s
Check the parameters for this Package.
$ tanzu package available get -n demo hello-nodejs.demo.tap/20230811080453.0.0+build.fde413c --values-schema
KEY DEFAULT TYPE DESCRIPTION
replicas 1 integer Number of replicas.
workload_name "" string Required. Name of the workload, used by K8s Ingress HTTP rules.
cluster_issuer tap-ingress-selfsigned string CertManager Issuer to use to generate certificate for K8s Ingress.
hostname "" string If set, K8s Ingress will be created with HTTP rules for hostname.
port 8080 integer Port number for the backend associated with K8s Ingress.
Set workload_name to hostname to use Ingress.
cat <<EOF > hello-nodejs-values.yaml
workload_name: hello-nodejs
hostname: hello-nodejs-demo.stg.192-168-228-200.sslip.io
EOF
Install this package with the following command: For multi cluster, run in Run cluster. By specifying >= 0.0.0 for -v, it will automatically update to the new version when a new package is created on Kubernetes.
tanzu package install -n demo hello-nodejs -p hello-nodejs.demo.tap -v ">= 0.0.0" --values-file hello-nodejs-values.yaml
After the installation is complete, you can see that the following resources have been created.
$ kubectl get pkgi,deploy,svc,ing -n demo
NAME PACKAGE NAME PACKAGE VERSION DESCRIPTION AGE
packageinstall.packaging.carvel.dev/hello-nodejs hello-nodejs.demo.tap 20230811080453.0.0+build.fde413c Reconcile succeeded 12s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/hello-nodejs 1/1 1 1 9s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/hello-nodejs ClusterIP 10.96.113.106 <none> 8080/TCP 9s
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/hello-nodejs <none> hello-nodejs-demo.stg.192-168-228-200.sslip.io 192.168.228.200 80, 443 9s
You can access URLs published via Ingress.
$ curl -k https://hello-nodejs-demo.stg.192-168-228-200.sslip.io
Hello World!!
In the case of Carvel Package Supply Chain, Ingress is created in addition to the resources (Deployment, Service) that are created in the normal Supply Chain type=server Workload.
Additional created resources are defined in overlay. This overlay file and modifiable parameter definitions can be overridden in tap-values.yaml since TAP 1.6.
The default definition is as follows.
ootb_templates:
carvel_package:
parameters:
- selector:
matchLabels:
apps.tanzu.vmware.com/workload-type: server
schema: |
#@data/values-schema
---
#@schema/title "Workload name"
#@schema/desc "Required. Name of the workload, used by K8s Ingress HTTP rules."
#@schema/example "tanzu-java-web-app"
#@schema/validation min_len=1
workload_name: ""
#@schema/title "Replicas"
#@schema/desc "Number of replicas."
replicas: 1
#@schema/title "Port"
#@schema/desc "Port number for the backend associated with K8s Ingress."
port: 8080
#@schema/title "Hostname"
#@schema/desc "If set, K8s Ingress will be created with HTTP rules for hostname."
#@schema/example "app.tanzu.vmware.com"
hostname: ""
#@schema/title "Cluster Issuer"
#@schema/desc "CertManager Issuer to use to generate certificate for K8s Ingress."
cluster_issuer: "tap-ingress-selfsigned"
overlays: |
#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")
#@overlay/match by=overlay.subset({"apiVersion":"apps/v1", "kind": "Deployment"})
---
spec:
#@overlay/match missing_ok=True
replicas: #@ data.values.replicas
#@ if data.values.hostname != "":
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: #@ data.values.workload_name
annotations:
cert-manager.io/cluster-issuer: #@ data.values.cluster_issuer
ingress.kubernetes.io/force-ssl-redirect: "true"
kubernetes.io/ingress.class: contour
kapp.k14s.io/change-rule: "upsert after upserting Services"
labels:
app.kubernetes.io/component: "run"
carto.run/workload-name: #@ data.values.workload_name
spec:
tls:
- secretName: #@ data.values.workload_name
hosts:
- #@ data.values.hostname
rules:
- host: #@ data.values.hostname
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: #@ data.values.workload_name
port:
number: #@ data.values.port
#@ end
Ingress has an annotation that uses Contour. Let's customize it to use Nginx Ingress as mentioned above.
Set the following in tap-values.yaml.
ootb_templates:
carvel_package:
parameters:
- selector:
matchLabels:
apps.tanzu.vmware.com/workload-type: server
schema: |
#@data/values-schema
---
#@schema/title "Workload name"
#@schema/desc "Required. Name of the workload, used by K8s Ingress HTTP rules."
#@schema/example "tanzu-java-web-app"
#@schema/validation min_len=1
workload_name: ""
#@schema/title "Replicas"
#@schema/desc "Number of replicas."
replicas: 1
#@schema/title "Port"
#@schema/desc "Port number for the backend associated with K8s Ingress."
port: 8080
#@schema/title "Hostname"
#@schema/desc "If set, K8s Ingress will be created with HTTP rules for hostname."
#@schema/example "app.tanzu.vmware.com"
hostname: ""
#@schema/title "Cluster Issuer"
#@schema/desc "CertManager Issuer to use to generate certificate for K8s Ingress."
cluster_issuer: "tap-ingress-selfsigned"
overlays: |
#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")
#@overlay/match by=overlay.subset({"apiVersion":"apps/v1", "kind": "Deployment"})
---
spec:
#@overlay/match missing_ok=True
replicas: #@ data.values.replicas
#@ if data.values.hostname != "":
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: #@ data.values.workload_name
annotations:
cert-manager.io/cluster-issuer: #@ data.values.cluster_issuer
ingress.kubernetes.io/force-ssl-redirect: "true"
#! kubernetes.io/ingress.class: contour <----- Removed
kapp.k14s.io/change-rule: "upsert after upserting Services"
labels:
app.kubernetes.io/component: "run"
carto.run/workload-name: #@ data.values.workload_name
spec:
ingressClassName: nginx #! <----- Added
tls:
- secretName: #@ data.values.workload_name
hosts:
- #@ data.values.hostname
rules:
- host: #@ data.values.hostname
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: #@ data.values.workload_name
port:
number: #@ data.values.port
#@ end
Update TAP with the following command:
tanzu package installed update -n tap-install tap --values-file tap-values.yaml
If you check the status of the workload, you can see that hello-nodejs-carvel-package-**** and hello-nodejs-pkg-cfg-writer-**** are newly generated.
A template change was detected and a new manifest imgpkg bundle created and git pushed.
$ tanzu apps workload get hello-nodejs --namespace demo
📡 Overview
name: hello-nodejs
type: server
namespace: demo
💾 Source
type: git
url: https://github.com/making/hello-nodejs
branch: master
revision: master@sha1:fde413c0fba0003c218a60bde69c8e254d3b15a6
📦 Supply Chain
name: source-to-url-package
NAME READY HEALTHY UPDATED RESOURCE
source-provider True True 9m41s gitrepositories.source.toolkit.fluxcd.io/hello-nodejs
image-provider True True 8m45s images.kpack.io/hello-nodejs
config-provider True True 8m38s podintents.conventions.carto.run/hello-nodejs
app-config True True 8m38s configmaps/hello-nodejs-server
service-bindings True True 8m38s configmaps/hello-nodejs-with-claims
api-descriptors True True 8m38s configmaps/hello-nodejs-with-api-descriptors
carvel-package True True 11s taskruns.tekton.dev/hello-nodejs-carvel-package-v7psj
config-writer Unknown Unknown 4s runnables.carto.run/hello-nodejs-pkg-cfg-writer
🚚 Delivery
Delivery resources not found.
💬 Messages
Workload [HealthyConditionRule]: Not all Steps in the Task have finished executing
🛶 Pods
NAME READY STATUS RESTARTS AGE
hello-nodejs-7455c98c94-c4zlt 1/1 Running 0 4m23s
hello-nodejs-build-1-build-pod 0/1 Completed 0 9m44s
hello-nodejs-carvel-package-j86xm-pod 0/3 Completed 0 8m39s
hello-nodejs-carvel-package-v7psj-pod 0/3 Completed 0 26s
hello-nodejs-pkg-cfg-writer-ctxfx-pod 0/2 Completed 0 13s
hello-nodejs-pkg-cfg-writer-jsg9x-pod 0/2 Completed 0 8m23s
To see logs: "tanzu apps workload tail hello-nodejs --namespace demo --timestamp --since 1h"
The generated Package resource is https://github.com/making/tap-gitops-manifests/blob/main/hello-nodejs.demo.tap/packages/20230811081306.0.0.yml.
You can download the bundled manifest with the command:
imgpkg pull -b ghcr.io/making/workloads/hello-nodejs-demo-bundle@sha256:8971d6979d6823edbe502162ddb9ab6dde665ac807857dbe4ea89857ce2db28c -o /tmp/hello-nodejs
Apply the new Package with kubectl manually. If you are using GitOps (recommended) the new Package will be added automatically.
kubectl apply -f https://github.com/making/tap-gitops-manifests/raw/main/hello-nodejs.demo.tap/packages/20230811081306.0.0.yml -n demo
You can check the new Package with the following command:
$ kubectl get package -n demo
NAME PACKAGEMETADATA NAME VERSION AGE
hello-nodejs.demo.tap.20230811080453.0.0+build.fde413c hello-nodejs.demo.tap 20230811080453.0.0+build.fde413c 5m53s
hello-nodejs.demo.tap.20230811081306.0.0+build.fde413c hello-nodejs.demo.tap 20230811081306.0.0+build.fde413c 6s
PackageInstall will detect new packages and install them automatically. The following command shows that a new version of the package has been deployed and that Ingress nginx is being used.
$ kubectl get pkgi,deploy,svc,ing -n demo
NAME PACKAGE NAME PACKAGE VERSION DESCRIPTION AGE
packageinstall.packaging.carvel.dev/hello-nodejs hello-nodejs.demo.tap 20230811081306.0.0+build.fde413c Reconcile succeeded 5m44s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/hello-nodejs 1/1 1 1 5m41s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/hello-nodejs ClusterIP 10.96.113.106 <none> 8080/TCP 5m41s
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/hello-nodejs nginx hello-nodejs-demo.stg.192-168-228-200.sslip.io 192.168.228.201 80, 443 5m41s
To use Routing via Nginx Ingress, change the hostname as follows.
cat <<EOF > hello-nodejs-values.yaml
workload_name: hello-nodejs
hostname: hello-nodejs-demo.stg.192-168-228-201.sslip.io
EOF
Reflect the new parameters with the following command:
tanzu package installed update -n demo hello-nodejs --values-file hello-nodejs-values.yaml
Confirm that the Ingress hostname has been updated with the following command:
$ kubectl get pkgi,deploy,svc,ing -n demo
NAME PACKAGE NAME PACKAGE VERSION DESCRIPTION AGE
packageinstall.packaging.carvel.dev/hello-nodejs hello-nodejs.demo.tap 20230811081306.0.0+build.fde413c Reconcile succeeded 6m26s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/hello-nodejs 1/1 1 1 6m23s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/hello-nodejs ClusterIP 10.96.113.106 <none> 8080/TCP 6m23s
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/hello-nodejs nginx hello-nodejs-demo.stg.192-168-228-201.sslip.io 192.168.228.201 80, 443 6m23s
Access the app using the new hostname.
$ curl -k https://hello-nodejs-demo.stg.192-168-228-201.sslip.io
Hello World!!
Using Carvel Package Supply Chain, I was able to test that parameters can be changed for each execution environment (Run cluster).
As of TAP 1.6, setting the execution environment feels a little complicated.
I introduced 3 ways to publish the app in Ingress when deploying the app with type=server.
- Create an Ingress resource directly with
kubectl - Add Ingress to resources created in OOTB SupplyChain
- Use Carvel Package Supply Chain
As of TAP 1.6, they all have pros and cons. When Carvel Package Supply Chain becomes GA, this may be the best choice.
At the moment, it may be more pragmatic to create Ingress resources directly with kubectl or add Ingress to resources created by OOTB SupplyChain.