--- title: Tanzu Kubernetes Grid on vSphereを1.2.0 -> 1.2.1にアップデートするメモ tags: ["Kubernetes", "vSphere", "TKG", "Tanzu", "Cluster API"] categories: ["Dev", "CaaS", "Kubernetes", "TKG", "vSphere"] date: 2020-12-11T16:21:27Z updated: 2020-12-11T16:28:29Z --- "[Tanzu Kubernetes Grid 1.2をvSphere 7にインストールするメモ](/entries/625)"で作成した環境をアップデートします。 現在TKG 1.2.0で次のようなクラスタを管理している状態で1.2.1にバージョンアップします。 ``` $ tkg get clusters --include-management-cluster NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES grape default running 1/1 3/3 v1.19.1+vmware.2 lemon default running 1/1 3/3 v1.19.1+vmware.2 carrot tkg-system running 1/1 1/1 v1.19.1+vmware.2 management ``` 念のため、`~/.tkg`ディレクトリをバックアップしておきます。 ``` cp -r ~/.tkg ~/.tkg-20201212 ``` アップデートの手順はこちら https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.2/vmware-tanzu-kubernetes-grid-12/GUID-upgrade-tkg-index.html **Table of contents** ### CLIのインストール [https://my.vmware.com/group/vmware/downloads/details?downloadGroup=TKG-121&productId=988&rPId=56777](https://my.vmware.com/group/vmware/downloads/details?downloadGroup=TKG-121&productId=988&rPId=56777) から - VMware Tanzu Kubernetes Grid CLI for Mac をダウンロードして次のようにインストールしてください。 ``` tar xzvf tkg-darwin-amd64-v1.2.1-vmware.1.tar.gz mv tkg/tkg-darwin-amd64-v1.2.1+vmware.1 /usr/local/bin/tkg chmod +x /usr/local/bin/tkg ``` 次のバージョンで動作確認しています。 ``` $ tkg version Client: Version: v1.2.1 Git commit: 9d15a485f2ccc462622f8df6a81e5fa831c51895 ``` 次のコマンドを実行すると1.2.1用の設定ファイルが追加されます。 ``` $ tkg get management-cluster It seems that the TKG settings on this system are out-of-date. Proceeding on this command will cause them to be backed up and overwritten by the latest settings. Do you want to continue? [y/N]: y the old providers folder /Users/toshiaki/.tkg/providers is backed up to /Users/toshiaki/.tkg/providers-20201212002337-ml4ooek8 The old bom folder /Users/toshiaki/.tkg/bom is backed up to /Users/toshiaki/.tkg/bom-20201212002337-j81lmhje MANAGEMENT-CLUSTER-NAME CONTEXT-NAME STATUS carrot * carrot-admin@carrot Success ``` 次のようなフォルダ構成になります。 ``` $ find ~/.tkg | grep -v 20201212002337 /Users/toshiaki/.tkg /Users/toshiaki/.tkg/bom /Users/toshiaki/.tkg/bom/bom-1.1.2+vmware.1.yaml /Users/toshiaki/.tkg/bom/bom-1.18.8+vmware.1.yaml /Users/toshiaki/.tkg/bom/bom-1.18.10+vmware.1.yaml /Users/toshiaki/.tkg/bom/bom-tkg-1.0.0.yaml /Users/toshiaki/.tkg/bom/bom-1.2.0+vmware.1.yaml /Users/toshiaki/.tkg/bom/bom-1.17.13+vmware.1.yaml /Users/toshiaki/.tkg/bom/bom-1.17.6+vmware.1.yaml /Users/toshiaki/.tkg/bom/bom-1.2.1+vmware.1.yaml /Users/toshiaki/.tkg/bom/bom-1.17.11+vmware.1.yaml /Users/toshiaki/.tkg/bom/bom-1.1.0+vmware.1.yaml /Users/toshiaki/.tkg/bom/bom-1.17.9+vmware.1.yaml /Users/toshiaki/.tkg/bom/bom-1.1.3+vmware.1.yaml /Users/toshiaki/.tkg/features.json /Users/toshiaki/.tkg/providers /Users/toshiaki/.tkg/providers/infrastructure-azure /Users/toshiaki/.tkg/providers/infrastructure-azure/v0.4.8 /Users/toshiaki/.tkg/providers/infrastructure-azure/v0.4.8/infrastructure-components.yaml /Users/toshiaki/.tkg/providers/infrastructure-azure/v0.4.8/cluster-template-definition-prod.yaml /Users/toshiaki/.tkg/providers/infrastructure-azure/v0.4.8/cluster-template-definition-dev.yaml /Users/toshiaki/.tkg/providers/infrastructure-azure/v0.4.8/ytt /Users/toshiaki/.tkg/providers/infrastructure-azure/v0.4.8/ytt/overlay.yaml /Users/toshiaki/.tkg/providers/infrastructure-azure/v0.4.8/ytt/base-template.yaml /Users/toshiaki/.tkg/providers/infrastructure-azure/ytt /Users/toshiaki/.tkg/providers/infrastructure-azure/ytt/azure-overlay.yaml /Users/toshiaki/.tkg/providers/control-plane-kubeadm /Users/toshiaki/.tkg/providers/control-plane-kubeadm/v0.3.11 /Users/toshiaki/.tkg/providers/control-plane-kubeadm/v0.3.11/control-plane-components.yaml /Users/toshiaki/.tkg/providers/providers.md5sum /Users/toshiaki/.tkg/providers/infrastructure-vsphere /Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1 /Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1/infrastructure-components.yaml /Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1/cluster-template-definition-prod.yaml /Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1/cluster-template-definition-dev.yaml /Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1/ytt /Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1/ytt/overlay.yaml /Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1/ytt/add_csi.yaml /Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1/ytt/csi-vsphere.lib.txt /Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1/ytt/csi.lib.yaml /Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1/ytt/base-template.yaml /Users/toshiaki/.tkg/providers/infrastructure-vsphere/ytt /Users/toshiaki/.tkg/providers/infrastructure-vsphere/ytt/vsphere-overlay.yaml /Users/toshiaki/.tkg/providers/cluster-api /Users/toshiaki/.tkg/providers/cluster-api/v0.3.11 /Users/toshiaki/.tkg/providers/cluster-api/v0.3.11/core-components.yaml /Users/toshiaki/.tkg/providers/config.yaml /Users/toshiaki/.tkg/providers/bootstrap-kubeadm /Users/toshiaki/.tkg/providers/bootstrap-kubeadm/v0.3.11 /Users/toshiaki/.tkg/providers/bootstrap-kubeadm/v0.3.11/bootstrap-components.yaml /Users/toshiaki/.tkg/providers/infrastructure-aws /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.5.5 /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.5.5/infrastructure-components.yaml /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.5.5/cluster-template-definition-prod.yaml /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.5.5/cluster-template-definition-dev.yaml /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.5.5/ytt /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.5.5/ytt/overlay.yaml /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.5.5/ytt/base-template.yaml /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.3 /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.3/infrastructure-components.yaml /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.3/metadata.yaml /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.3/cluster-template-definition-prod.yaml /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.3/cluster-template-definition-dev.yaml /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.3/ytt /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.3/ytt/overlay.yaml /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.3/ytt/base-template.yaml /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.1 /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.1/infrastructure-components.yaml /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.1/metadata.yaml /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.1/cluster-template-definition-prod.yaml /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.1/cluster-template-definition-dev.yaml /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.1/ytt /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.1/ytt/overlay.yaml /Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.1/ytt/base-template.yaml /Users/toshiaki/.tkg/providers/infrastructure-aws/ytt /Users/toshiaki/.tkg/providers/infrastructure-aws/ytt/aws-overlay.yaml /Users/toshiaki/.tkg/providers/config_default.yaml /Users/toshiaki/.tkg/providers/infrastructure-docker /Users/toshiaki/.tkg/providers/infrastructure-docker/v0.3.10 /Users/toshiaki/.tkg/providers/infrastructure-docker/v0.3.10/infrastructure-components.yaml /Users/toshiaki/.tkg/providers/infrastructure-docker/v0.3.10/metadata.yaml /Users/toshiaki/.tkg/providers/infrastructure-docker/v0.3.10/cluster-template-definition-prod.yaml /Users/toshiaki/.tkg/providers/infrastructure-docker/v0.3.10/cluster-template-definition-dev.yaml /Users/toshiaki/.tkg/providers/infrastructure-docker/v0.3.10/ytt /Users/toshiaki/.tkg/providers/infrastructure-docker/v0.3.10/ytt/overlay.yaml /Users/toshiaki/.tkg/providers/infrastructure-docker/v0.3.10/ytt/base-template.yaml /Users/toshiaki/.tkg/providers/infrastructure-docker/ytt /Users/toshiaki/.tkg/providers/infrastructure-docker/ytt/docker-overlay.yaml /Users/toshiaki/.tkg/providers/infrastructure-tkg-service-vsphere /Users/toshiaki/.tkg/providers/infrastructure-tkg-service-vsphere/v1.0.0 /Users/toshiaki/.tkg/providers/infrastructure-tkg-service-vsphere/v1.0.0/cluster-template-definition-prod.yaml /Users/toshiaki/.tkg/providers/infrastructure-tkg-service-vsphere/v1.0.0/cluster-template-definition-dev.yaml /Users/toshiaki/.tkg/providers/infrastructure-tkg-service-vsphere/v1.0.0/ytt /Users/toshiaki/.tkg/providers/infrastructure-tkg-service-vsphere/v1.0.0/ytt/overlay.yaml /Users/toshiaki/.tkg/providers/infrastructure-tkg-service-vsphere/v1.0.0/ytt/base-template.yaml /Users/toshiaki/.tkg/providers/infrastructure-tkg-service-vsphere/ytt /Users/toshiaki/.tkg/providers/infrastructure-tkg-service-vsphere/ytt/tkg-service-vsphere-overlay.yaml /Users/toshiaki/.tkg/providers/ytt /Users/toshiaki/.tkg/providers/ytt/02_addons /Users/toshiaki/.tkg/providers/ytt/02_addons/cni /Users/toshiaki/.tkg/providers/ytt/02_addons/cni/antrea /Users/toshiaki/.tkg/providers/ytt/02_addons/cni/antrea/antrea.lib.yaml /Users/toshiaki/.tkg/providers/ytt/02_addons/cni/antrea/antrea_overlay.lib.yaml /Users/toshiaki/.tkg/providers/ytt/02_addons/cni/add_cni.yaml /Users/toshiaki/.tkg/providers/ytt/02_addons/cni/calico /Users/toshiaki/.tkg/providers/ytt/02_addons/cni/calico/calico_overlay.lib.yaml /Users/toshiaki/.tkg/providers/ytt/02_addons/cni/calico/calico.lib.yaml /Users/toshiaki/.tkg/providers/ytt/02_addons/metadata /Users/toshiaki/.tkg/providers/ytt/02_addons/metadata/add_cluster_metadata.yaml /Users/toshiaki/.tkg/providers/ytt/01_plans /Users/toshiaki/.tkg/providers/ytt/01_plans/dev.yaml /Users/toshiaki/.tkg/providers/ytt/01_plans/prod.yaml /Users/toshiaki/.tkg/providers/ytt/01_plans/oidc.yaml /Users/toshiaki/.tkg/providers/ytt/03_customizations /Users/toshiaki/.tkg/providers/ytt/03_customizations/registry_ca_cert.yaml /Users/toshiaki/.tkg/providers/ytt/03_customizations/filter.yaml /Users/toshiaki/.tkg/providers/ytt/03_customizations/add_mhc.yaml /Users/toshiaki/.tkg/providers/ytt/03_customizations/add_sc.yaml /Users/toshiaki/.tkg/providers/ytt/03_customizations/registry_skip_tls_verify.yaml /Users/toshiaki/.tkg/providers/ytt/03_customizations/autoscaler /Users/toshiaki/.tkg/providers/ytt/03_customizations/autoscaler/enable_autoscaler.yaml /Users/toshiaki/.tkg/providers/ytt/03_customizations/autoscaler/autoscaler_overlay.yaml /Users/toshiaki/.tkg/providers/ytt/lib /Users/toshiaki/.tkg/providers/ytt/lib/validate.star /Users/toshiaki/.tkg/providers/ytt/lib/helpers.star /Users/toshiaki/.tkg/.providers.zip.lock /Users/toshiaki/.tkg/config.yaml /Users/toshiaki/.tkg/.boms.lock ``` サポートされるKubernetesのバージョンは次の通りです。 ``` $ tkg get kubernetesversions VERSIONS v1.17.11+vmware.1 v1.17.13+vmware.1 v1.17.3+vmware.2 v1.17.6+vmware.1 v1.17.9+vmware.1 v1.18.10+vmware.1 v1.18.2+vmware.1 v1.18.3+vmware.1 v1.18.6+vmware.1 v1.18.8+vmware.1 v1.19.1+vmware.2 v1.19.3+vmware.1 ``` ### vCenterにOVAファイルをアップロード [https://my.vmware.com/en/group/vmware/downloads/details?downloadGroup=TKG-120&productId=988&rPId=48121](https://my.vmware.com/en/group/vmware/downloads/details?downloadGroup=TKG-120&productId=988&rPId=48121) から - Photon v3 Kubernetes v1.19.3 OVA をダウンロードし、`~/Downloads`に保存します。1.18.10を使いたい場合も以下同様です。 TKGをインストールする環境の情報を次の例のように設定してください。インストール先のネットワークは**DHCPが有効になっている必要があります**。 ``` cat < tkg-env.sh export GOVC_URL=vcsa.v.maki.lol export GOVC_USERNAME=administrator@vsphere.local export GOVC_PASSWORD=VMware1! export GOVC_DATACENTER=Datacenter export GOVC_NETWORK="VM Network" export GOVC_DATASTORE=datastore1 export GOVC_RESOURCE_POOL=/Datacenter/host/Cluster/Resources/tkg export GOVC_INSECURE=1 export TEMPLATE_FOLDER=/Datacenter/vm/tkg EOF ``` 次のコマンドでOVAファイルをvCenterにアップロードします。 ``` source tkg-env.sh govc import.ova -folder $TEMPLATE_FOLDER ~/Downloads/photon-3-kube-v1.19.3-vmware.1.ova govc vm.markastemplate $TEMPLATE_FOLDER/photon-3-kube-v1.19.3 ``` ### Management Clusterをアップデート 次のコマンドでManagement Clusterをアップデートします。 ``` tkg upgrade management-cluster carrot ``` 次のようなログが出力されます。 ``` Logs of the command execution can also be found at: /var/folders/76/vg4pyy253pbgwzncmx2mb1gh0000gq/T/tkg-20201212T004546901230328.log Upgrading management cluster providers... Checking cert-manager version... Cert-manager is already up to date Performing upgrade... Deleting Provider="cluster-api" Version="" TargetNamespace="capi-system" Installing Provider="cluster-api" Version="v0.3.11" TargetNamespace="capi-system" Deleting Provider="bootstrap-kubeadm" Version="" TargetNamespace="capi-kubeadm-bootstrap-system" Installing Provider="bootstrap-kubeadm" Version="v0.3.11" TargetNamespace="capi-kubeadm-bootstrap-system" Deleting Provider="control-plane-kubeadm" Version="" TargetNamespace="capi-kubeadm-control-plane-system" Installing Provider="control-plane-kubeadm" Version="v0.3.11" TargetNamespace="capi-kubeadm-control-plane-system" Deleting Provider="infrastructure-vsphere" Version="" TargetNamespace="capv-system" Installing Provider="infrastructure-vsphere" Version="v0.7.1" TargetNamespace="capv-system" Management cluster providers upgraded successfully... Upgrading management cluster kubernetes version... Verifying kubernetes version... Retrieving configuration for upgrade cluster... consuming Azure VM image information from BOM Create InfrastructureTemplate for upgrade... Upgrading control plane nodes... Patching KubeadmControlPlane with the kubernetes version v1.19.3+vmware.1... Waiting for kubernetes version to be updated for control plane nodes Upgrading worker nodes... Patching MachineDeployment with the kubernetes version v1.19.3+vmware.1... Waiting for kubernetes version to be updated for worker nodes... updating 'metadata/tkg' add-on... Management cluster 'carrot' successfully upgraded to TKG version 'v1.2.1' with kubernetes version 'v1.19.3+vmware.1' ... ``` 次のコマンドでManagment ClusterのみKubernetes 1.19.3にバージョンアップされたことがわかります。 ``` $ tkg get cluster --include-management-cluster NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES grape default running 1/1 3/3 v1.19.1+vmware.2 lemon default running 1/1 3/3 v1.19.1+vmware.2 carrot tkg-system running 1/1 1/1 v1.19.3+vmware.1 management ``` Management Clusterに`kubectl`コマンドでアクセスしてみます。 ``` $ kubectl cluster-info Kubernetes master is running at https://192.168.11.110:6443 KubeDNS is running at https://192.168.11.110:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. $ kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME carrot-control-plane-9bbxq Ready master 11m v1.19.3+vmware.1 192.168.11.97 192.168.11.97 VMware Photon OS/Linux 4.19.150-1.ph3 containerd://1.4.1 carrot-md-0-647c46f976-9cnsb Ready 5m41s v1.19.3+vmware.1 192.168.11.98 192.168.11.98 VMware Photon OS/Linux 4.19.150-1.ph3 containerd://1.4.1 ``` ### Workload Clusterをアップデート 対象のManagement Clusterを`tkg set management-cluster`で指定します。1つしかなければ不要です。 ``` $ tkg set management-cluster carrot The current management cluster context is switched to carrot ``` 対象となっているManagement Clusterを`tkg set management-cluster`で確認できます。`*`がついているクラスタが対象です。 `tkg update cluster`でWorkload Clusterを作成します。 ``` tkg upgrade cluster grape --yes ``` 次のようなログが出力されます。 ``` Logs of the command execution can also be found at: /var/folders/76/vg4pyy253pbgwzncmx2mb1gh0000gq/T/tkg-20201212T010537187656254.log Validating configuration... Verifying kubernetes version... Retrieving configuration for upgrade cluster... consuming Azure VM image information from BOM Create InfrastructureTemplate for upgrade... Upgrading control plane nodes... Patching KubeadmControlPlane with the kubernetes version v1.19.3+vmware.1... Waiting for kubernetes version to be updated for control plane nodes Upgrading worker nodes... Patching MachineDeployment with the kubernetes version v1.19.3+vmware.1... Waiting for kubernetes version to be updated for worker nodes... updating 'metadata/tkg' add-on... Cluster 'grape' successfully upgraded to kubernetes version 'v1.19.3+vmware.1' ``` `tkg get cluster`でWorkload Clusterを確認できます。`grape`のみKubernetes 1.19.3にバージョンアップされたことがわかります ``` $ tkg get cluster NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES grape default running 1/1 3/3 v1.19.3+vmware.1 lemon default running 1/1 3/3 v1.19.1+vmware.2 ``` Workload Cluster `grape`を確認します。 ``` $ kubectl config use-context grape-admin@grape $ kubectl cluster-info Kubernetes master is running at https://192.168.11.113:6443 KubeDNS is running at https://192.168.11.113:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. $ kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME grape-control-plane-st8sh Ready master 11m v1.19.3+vmware.1 192.168.11.30 192.168.11.30 VMware Photon OS/Linux 4.19.150-1.ph3 containerd://1.4.1 grape-md-0-7486cf9b49-ft96f Ready 6m3s v1.19.3+vmware.1 192.168.11.48 192.168.11.48 VMware Photon OS/Linux 4.19.150-1.ph3 containerd://1.4.1 grape-md-0-7486cf9b49-hrsn8 Ready 4m45s v1.19.3+vmware.1 192.168.11.51 192.168.11.51 VMware Photon OS/Linux 4.19.150-1.ph3 containerd://1.4.1 grape-md-0-7486cf9b49-shfqn Ready 7m15s v1.19.3+vmware.1 192.168.11.31 192.168.11.31 VMware Photon OS/Linux 4.19.150-1.ph3 containerd://1.4.1 ``` 他のクラスタも同じ手順でアップデートできます。 --- TKG 1.2.0を1.2.1にアップデートしました。アップデートも簡単でした。