📝 BLOG.IK.AM

@making's memo
(🗃 Categories 🏷 Tags)

Kubernetesことはじめ (Google Container Engine編)

🗃 {Dev/CaaS/Kubernetes}

🏷 Google Container Engine 🏷 Kubernetes

🗓 Updated at 2017-06-06T17:55:40+09:00 by Toshiaki Maki  🗓 Created at 2017-06-05T13:01:40+09:00 by Toshiaki Maki  {✒️️ Edit  ⏰ History}


🚨 Warning: This content is old. Please don't trust too much.

前回、MinikubeでKubernetesを始めてみたが、今度は同じ内容をGoogle Container Engine (GKE)で試す。

目次

事前準備

gcloudコマンドはインストール済み。

$ gcloud --version
Google Cloud SDK 156.0.0
bq 2.0.24
core 2017.05.19
gcloud 
gsutil 4.26

k8sのバージョンはこちら。

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T20:41:07Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.7", GitCommit:"8eb75a5810cba92ccad845ca360cf924f2385881", GitTreeState:"clean", BuildDate:"2017-04-27T09:42:05Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

次の設定をしておく。

gcloud auth login
gcloud config set project <Project ID>
gcloud config set compute/zone asia-northeast1-a

Kubernetesクラスタの作成

とりあえずg1-smallな小さめのワーカーインスタンスを1つだけ作る。(デフォルトはn1-standard-1が3インスタンス)

$ gcloud container clusters create hello-kube --num-nodes 1 --machine-type g1-small

Creating cluster hello-kube...done.                                                                                                                                                                                          
Created [https://container.googleapis.com/v1/projects/myproject/zones/asia-northeast1-a/clusters/hello-kube].
kubeconfig entry generated for hello-kube.
NAME        ZONE               MASTER_VERSION  MASTER_IP       MACHINE_TYPE  NODE_VERSION  NUM_NODES  STATUS
hello-kube  asia-northeast1-a  1.5.7           xx.xx.xx.xx     g1-small      1.5.7         1          RUNNING

Compute Engine画面にGKE用のVMができている。

image

Container Engine画面にクラスターができている。

image

gcloudコマンドでも確認。

$ gcloud compute instances list
NAME                                       ZONE               MACHINE_TYPE               PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
gke-hello-kube-default-pool-29d9abc0-lhcm  asia-northeast1-a  g1-small                                10.146.0.2   xx.xxx.xx.xxx   RUNNING

kubectlのクラスタ情報がいつの間にか変わっていた。

$ kubectl cluster-info
Kubernetes master is running at https://xx.xxx.xxx.xxx
GLBCDefaultBackend is running at https://xx.xxx.xxx.xxx/api/v1/proxy/namespaces/kube-system/services/default-http-backend
Heapster is running at https://xx.xxx.xxx.xxx/api/v1/proxy/namespaces/kube-system/services/heapster
KubeDNS is running at https://xx.xxx.xxx.xxx/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at https://xx.xxx.xxx.xxx/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

DashboardのURLも記載されているが、アクセスするとBasic認証がかかっている。認証情報はクラスタ画面の"Show credentials"から確認できる。

image

メモ

各種クラスタ情報は~/.kube/configに保存されている。Minikubeに戻したい場合は、

kubectl config use-context minikube

を実行すれば良い。

CLIだけでアプリケーションをデプロイする

まずはYAMLなしでデプロイ。

Deploymentの作成。

kubectl run hello-tomcat --image=making/hello-tomcat:v1 --port=8080 

各種表示

$ kubectl get deployment -o wide
NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE       CONTAINER(S)   IMAGE(S)                 SELECTOR
hello-tomcat   1         1         1            1           41s       hello-tomcat   making/hello-tomcat:v1   run=hello-tomcat

$ kubectl get pod -o wide
NAME                            READY     STATUS    RESTARTS   AGE       IP         NODE
hello-tomcat-1638778051-6jwlx   1/1       Running   0          49s       10.0.0.9   gke-hello-kube-default-pool-29d9abc0-lhcm

$ kubectl get replicaset -o wide
NAME                      DESIRED   CURRENT   READY     AGE       CONTAINER(S)   IMAGE(S)                 SELECTOR
hello-tomcat-1638778051   1         1         1         1m        hello-tomcat   making/hello-tomcat:v1   pod-template-hash=1638778051,run=hello-tomcat

image

Deploymentの詳細表示

$ kubectl describe deployment hello-tomcat
Name:            hello-tomcat
Namespace:        default
CreationTimestamp:    Tue, 30 May 2017 01:58:43 +0900
Labels:            run=hello-tomcat
Annotations:        deployment.kubernetes.io/revision=1
Selector:        run=hello-tomcat
Replicas:        1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:        RollingUpdate
MinReadySeconds:    0
RollingUpdateStrategy:    1 max unavailable, 1 max surge
Pod Template:
  Labels:    run=hello-tomcat
  Containers:
   hello-tomcat:
    Image:        making/hello-tomcat:v1
    Port:        8080/TCP
    Environment:    <none>
    Mounts:        <none>
  Volumes:        <none>
Conditions:
  Type        Status    Reason
  ----        ------    ------
  Available     True    MinimumReplicasAvailable
OldReplicaSets:    <none>
NewReplicaSet:    <none>
Events:
  FirstSeen    LastSeen    Count    From            SubObjectPath    Type        Reason            Message
  ---------    --------    -----    ----            -------------    --------    ------            -------
  6m        6m        1    deployment-controller            Normal        ScalingReplicaSet    Scaled up replica set hello-tomcat-1638778051 to 1

Podの詳細表示

$ kubectl describe pod hello-tomcat-1638778051-6jwlx
Name:        hello-tomcat-1638778051-6jwlx
Namespace:    default
Node:        gke-hello-kube-default-pool-29d9abc0-lhcm/10.146.0.2
Start Time:    Tue, 30 May 2017 01:58:43 +0900
Labels:        pod-template-hash=1638778051
        run=hello-tomcat
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"hello-tomcat-1638778051","uid":"12aa7ff4-4490-11e7-8d34-42010a92...
        kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container hello-tomcat
Status:        Running
IP:        10.0.0.9
Controllers:    ReplicaSet/hello-tomcat-1638778051
Containers:
  hello-tomcat:
    Container ID:    docker://7d3175e71d2bf52ca2cad694729bd2032fe16fb58b61686fd2d0827d6598fd4f
    Image:        making/hello-tomcat:v1
    Image ID:        docker://sha256:a59c0230b53e29078b0725c35c8b64f3d276b31a59a0b8cd5f7226eec7fc2d7a
    Port:        8080/TCP
    State:        Running
      Started:        Tue, 30 May 2017 01:59:10 +0900
    Ready:        True
    Restart Count:    0
    Requests:
      cpu:        100m
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-c1k77 (ro)
Conditions:
  Type        Status
  Initialized     True 
  Ready     True 
  PodScheduled     True 
Volumes:
  default-token-c1k77:
    Type:    Secret (a volume populated by a Secret)
    SecretName:    default-token-c1k77
    Optional:    false
QoS Class:    Burstable
Node-Selectors:    <none>
Tolerations:    <none>
Events:
  FirstSeen    LastSeen    Count    From                            SubObjectPath            Type        Reason        Message
  ---------    --------    -----    ----                            -------------            --------    ------        -------
  9m        9m        1    default-scheduler                                    Normal        Scheduled    Successfully assigned hello-tomcat-1638778051-6jwlx to gke-hello-kube-default-pool-29d9abc0-lhcm
  9m        9m        1    kubelet, gke-hello-kube-default-pool-29d9abc0-lhcm    spec.containers{hello-tomcat}    Normal        Pulling        pulling image "making/hello-tomcat:v1"
  9m        9m        1    kubelet, gke-hello-kube-default-pool-29d9abc0-lhcm    spec.containers{hello-tomcat}    Normal        Pulled        Successfully pulled image "making/hello-tomcat:v1"
  9m        9m        1    kubelet, gke-hello-kube-default-pool-29d9abc0-lhcm    spec.containers{hello-tomcat}    Normal        Created        Created container with docker id 7d3175e71d2b; Security:[seccomp=unconfined]
  9m        9m        1    kubelet, gke-hello-kube-default-pool-29d9abc0-lhcm    spec.containers{hello-tomcat}    Normal        Started        Started container with docker id 7d3175e71d2b

Replicasetの詳細表示

$ kubectl describe replicaset hello-tomcat-1638778051
Name:        hello-tomcat-1638778051
Namespace:    default
Selector:    pod-template-hash=1638778051,run=hello-tomcat
Labels:        pod-template-hash=1638778051
        run=hello-tomcat
Annotations:    deployment.kubernetes.io/desired-replicas=1
        deployment.kubernetes.io/max-replicas=2
        deployment.kubernetes.io/revision=1
Replicas:    1 current / 1 desired
Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:    pod-template-hash=1638778051
        run=hello-tomcat
  Containers:
   hello-tomcat:
    Image:        making/hello-tomcat:v1
    Port:        8080/TCP
    Environment:    <none>
    Mounts:        <none>
  Volumes:        <none>
Events:
  FirstSeen    LastSeen    Count    From            SubObjectPath    Type        Reason            Message
  ---------    --------    -----    ----            -------------    --------    ------            -------
  10m        10m        1    replicaset-controller            Normal        SuccessfulCreate    Created pod: hello-tomcat-1638778051-6jwlx

コンテナの内部からcurlでWebアプリケーションにアクセス

$ kubectl exec -ti hello-tomcat-1638778051-6jwlx /bin/bash

root@hello-tomcat-1638778051-6jwlx:/usr/local/tomcat# curl -s http://localhost:8080/env | sort
CATALINA_HOME: /usr/local/tomcat
CA_CERTIFICATES_JAVA_VERSION: 20161107~bpo8+1
GPG_KEYS: 05AB33110949707C93A279E3D3EFE6B686867BA6 07E48665A34DCAFAE522E5E6266191C37C037D42 47309207D818FFD8DCD3F83F1931D684307A10A5 541FBE7D8F78B25E055DDEE13C370389288584E7 61B832AC2F1C5A90F0F9B00A1C506407564C17A3 713DA88BE50911535FE716F5208B0AB1D63011C7 79F7026C690BAA50B92CD8B66A3AD3F4F22C4FED 9BA44C2621385CB966EBA586F72C284D731FABEE A27677289986DB50844682F8ACB77FC2E86E29AC A9C5DF4D22E99998D9875A5110C01C5A2F6059E7 DCFD35E0BF8CA7344752DE8B6FB21E8933C60243 F3A04C595DB5B6A5F1ECA43E3B7BBB100D811BBE F7DA48BB64BCB84ECBA7EE6935CD23C10D498E23
HOME: /root
HOSTNAME: hello-tomcat-1638778051-6jwlx
JAVA_DEBIAN_VERSION: 8u131-b11-1~bpo8+1
JAVA_HOME: /docker-java-home/jre
JAVA_VERSION: 8u131
KUBERNETES_PORT: tcp://10.3.240.1:443
KUBERNETES_PORT_443_TCP: tcp://10.3.240.1:443
KUBERNETES_PORT_443_TCP_ADDR: 10.3.240.1
KUBERNETES_PORT_443_TCP_PORT: 443
KUBERNETES_PORT_443_TCP_PROTO: tcp
KUBERNETES_SERVICE_HOST: 10.3.240.1
KUBERNETES_SERVICE_PORT: 443
KUBERNETES_SERVICE_PORT_HTTPS: 443
LANG: C.UTF-8
LD_LIBRARY_PATH: /usr/local/tomcat/native-jni-lib
OPENSSL_VERSION: 1.1.0e-2
PATH: /usr/local/tomcat/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD: /usr/local/tomcat
TOMCAT_ASC_URL: https://www.apache.org/dist/tomcat/tomcat-8/v8.5.15/bin/apache-tomcat-8.5.15.tar.gz.asc
TOMCAT_MAJOR: 8
TOMCAT_NATIVE_LIBDIR: /usr/local/tomcat/native-jni-lib
TOMCAT_TGZ_URL: https://www.apache.org/dyn/closer.cgi?action=download&filename=tomcat/tomcat-8/v8.5.15/bin/apache-tomcat-8.5.15.tar.gz
TOMCAT_VERSION: 8.5.15

Serviceの作成(外部トラフィックの許可)

外部トラフィックを受け付けるためのServiceを作成する。

kubectl expose deployment hello-tomcat --type=LoadBalancer

Serviceの情報を見る。

$ kubectl get service hello-tomcat -o wide
NAME           CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE       SELECTOR
hello-tomcat   10.3.253.90   <pending>     8080:30085/TCP   10s       run=hello-tomcat

しばらくするとEXTERNAL-IPが割り振られる。

$ kubectl get service hello-tomcat -o wide
NAME           CLUSTER-IP    EXTERNAL-IP    PORT(S)          AGE       SELECTOR
hello-tomcat   10.3.253.90   yy.yyy.yy.yy   8080:30085/TCP   50s       run=hello-tomcat

詳細表示。

$ kubectl describe service hello-tomcat
Name:            hello-tomcat
Namespace:        default
Labels:            run=hello-tomcat
Annotations:        <none>
Selector:        run=hello-tomcat
Type:            LoadBalancer
IP:            10.3.253.90
LoadBalancer Ingress:    yy.yyy.yy.yy
Port:            <unset>    8080/TCP
NodePort:        <unset>    30085/TCP
Endpoints:        10.0.0.9:8080
Session Affinity:    None
Events:
  FirstSeen    LastSeen    Count    From            SubObjectPath    Type        Reason            Message
  ---------    --------    -----    ----            -------------    --------    ------            -------
  2m        2m        1    service-controller            Normal        CreatingLoadBalancer    Creating load balancer
  1m        1m        1    service-controller            Normal        CreatedLoadBalancer    Created load balancer

GCPのロードバランサが作成されている。

image

ロードバランスのIP:8080でWebアプリケーションにアクセスできる。

$ curl -s yy.yyy.yy.yy:8080/env | sort
CATALINA_HOME: /usr/local/tomcat
CA_CERTIFICATES_JAVA_VERSION: 20161107~bpo8+1
GPG_KEYS: 05AB33110949707C93A279E3D3EFE6B686867BA6 07E48665A34DCAFAE522E5E6266191C37C037D42 47309207D818FFD8DCD3F83F1931D684307A10A5 541FBE7D8F78B25E055DDEE13C370389288584E7 61B832AC2F1C5A90F0F9B00A1C506407564C17A3 713DA88BE50911535FE716F5208B0AB1D63011C7 79F7026C690BAA50B92CD8B66A3AD3F4F22C4FED 9BA44C2621385CB966EBA586F72C284D731FABEE A27677289986DB50844682F8ACB77FC2E86E29AC A9C5DF4D22E99998D9875A5110C01C5A2F6059E7 DCFD35E0BF8CA7344752DE8B6FB21E8933C60243 F3A04C595DB5B6A5F1ECA43E3B7BBB100D811BBE F7DA48BB64BCB84ECBA7EE6935CD23C10D498E23
HOME: /root
HOSTNAME: hello-tomcat-1638778051-6jwlx
JAVA_DEBIAN_VERSION: 8u131-b11-1~bpo8+1
JAVA_HOME: /docker-java-home/jre
JAVA_VERSION: 8u131
KUBERNETES_PORT: tcp://10.3.240.1:443
KUBERNETES_PORT_443_TCP: tcp://10.3.240.1:443
KUBERNETES_PORT_443_TCP_ADDR: 10.3.240.1
KUBERNETES_PORT_443_TCP_PORT: 443
KUBERNETES_PORT_443_TCP_PROTO: tcp
KUBERNETES_SERVICE_HOST: 10.3.240.1
KUBERNETES_SERVICE_PORT: 443
KUBERNETES_SERVICE_PORT_HTTPS: 443
LANG: C.UTF-8
LD_LIBRARY_PATH: /usr/local/tomcat/native-jni-lib
OPENSSL_VERSION: 1.1.0e-2
PATH: /usr/local/tomcat/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD: /usr/local/tomcat
TOMCAT_ASC_URL: https://www.apache.org/dist/tomcat/tomcat-8/v8.5.15/bin/apache-tomcat-8.5.15.tar.gz.asc
TOMCAT_MAJOR: 8
TOMCAT_NATIVE_LIBDIR: /usr/local/tomcat/native-jni-lib
TOMCAT_TGZ_URL: https://www.apache.org/dyn/closer.cgi?action=download&filename=tomcat/tomcat-8/v8.5.15/bin/apache-tomcat-8.5.15.tar.gz
TOMCAT_VERSION: 8.5.15

スケールアウト

kubectl scale --replicas=2 deployment/hello-tomcat

メモ

デフォルトのリソース設定だと、g1-smallインスタンスの場合、3インスタンスはリソース不足になる

各種表示

$ kubectl get deployment hello-tomcat -o wide
NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE       CONTAINER(S)   IMAGE(S)                 SELECTOR
hello-tomcat   2         2         2            2           29m       hello-tomcat   making/hello-tomcat:v1   run=hello-tomcat

$ kubectl get pod -l run=hello-tomcat -o wide
NAME                            READY     STATUS    RESTARTS   AGE       IP          NODE
hello-tomcat-1638778051-6jwlx   1/1       Running   0          29m       10.0.0.9    gke-hello-kube-default-pool-29d9abc0-lhcm
hello-tomcat-1638778051-lqnqm   1/1       Running   0          5m        10.0.0.10   gke-hello-kube-default-pool-29d9abc0-lhcm

$ kubectl get replicaset -l run=hello-tomcat -o wide
NAME                      DESIRED   CURRENT   READY     AGE       CONTAINER(S)   IMAGE(S)                 SELECTOR
hello-tomcat-1638778051   2         2         2         30m       hello-tomcat   making/hello-tomcat:v1   pod-template-hash=1638778051,run=hello-tomcat

ServiceのEntpointsが2つに増えている。

$ kubectl describe service hello-tomcat
Name:            hello-tomcat
Namespace:        default
Labels:            run=hello-tomcat
Annotations:        <none>
Selector:        run=hello-tomcat
Type:            LoadBalancer
IP:            10.3.253.90
LoadBalancer Ingress:    yy.yyy.yy.yy
Port:            <unset>    8080/TCP
NodePort:        <unset>    30085/TCP
Endpoints:        10.0.0.10:8080,10.0.0.9:8080
Session Affinity:    None
Events:
  FirstSeen    LastSeen    Count    From            SubObjectPath    Type        Reason            Message
  ---------    --------    -----    ----            -------------    --------    ------            -------
  17m        17m        1    service-controller            Normal        CreatingLoadBalancer    Creating load balancer
  16m        16m        1    service-controller            Normal        CreatedLoadBalancer    Created load balancer

ロードバランスの確認。

$ for i in `seq 1 10`;do curl -s yy.yyy.yy.yy:8080/env | grep HOSTNAME;done
HOSTNAME: hello-tomcat-1638778051-lqnqm
HOSTNAME: hello-tomcat-1638778051-lqnqm
HOSTNAME: hello-tomcat-1638778051-lqnqm
HOSTNAME: hello-tomcat-1638778051-6jwlx
HOSTNAME: hello-tomcat-1638778051-lqnqm
HOSTNAME: hello-tomcat-1638778051-6jwlx
HOSTNAME: hello-tomcat-1638778051-6jwlx
HOSTNAME: hello-tomcat-1638778051-6jwlx
HOSTNAME: hello-tomcat-1638778051-6jwlx
HOSTNAME: hello-tomcat-1638778051-6jwlx

ServiceとDeploymentの削除

kubectl delete service hello-tomcat
kubectl delete deployment hello-tomcat

YAMLを使ったデプロイ

普通はYAMLを使ってデプロイする。

次のようなYAMLを作って

kind: Service
apiVersion: v1
metadata:
  name: hello-tomcat-service
spec:
  selector:
    run: hello-tomcat
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  type: LoadBalancer
---
apiVersion: extensions/v1beta1 # k8s 1.6の場合はapps/v1beta1
kind: Deployment
metadata:
  name: hello-tomcat-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        run: hello-tomcat
    spec:
      containers:
      - name: hello-tomcat
        image: making/hello-tomcat:v1
        ports:
        - containerPort: 8080

を作って

kubectl create -f hello-tomcat.yml

各種表示

$ kubectl get pod -o wide
NAME                                       READY     STATUS    RESTARTS   AGE       IP          NODE
hello-tomcat-deployment-1638778051-rltr5   1/1       Running   0          31s       10.0.0.13   gke-hello-kube-default-pool-29d9abc0-lhcm
hello-tomcat-deployment-1638778051-wkx0q   1/1       Running   0          31s       10.0.0.14   gke-hello-kube-default-pool-29d9abc0-lhcm

$ kubectl get deployment hello-tomcat-deployment -o wide
NAME                      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE       CONTAINER(S)   IMAGE(S)              SELECTOR
hello-tomcat-deployment   2         2         2            2           42s       hello-tomcat   making/hello-tomcat:v1   run=hello-tomcat

$ kubectl get service hello-tomcat-service -o wide
NAME                   CLUSTER-IP     EXTERNAL-IP     PORT(S)          AGE       SELECTOR
hello-tomcat-service   10.3.242.224   zz.zzz.zz.zzz   8080:31599/TCP   56s       run=hello-tomcat

ロードバランスの確認。

$ for i in `seq 1 10`;do curl -s zz.zzz.zz.zzz:8080/env | grep HOSTNAME;done
HOSTNAME: hello-tomcat-deployment-1638778051-wkx0q
HOSTNAME: hello-tomcat-deployment-1638778051-rltr5
HOSTNAME: hello-tomcat-deployment-1638778051-rltr5
HOSTNAME: hello-tomcat-deployment-1638778051-wkx0q
HOSTNAME: hello-tomcat-deployment-1638778051-wkx0q
HOSTNAME: hello-tomcat-deployment-1638778051-rltr5
HOSTNAME: hello-tomcat-deployment-1638778051-rltr5
HOSTNAME: hello-tomcat-deployment-1638778051-rltr5
HOSTNAME: hello-tomcat-deployment-1638778051-rltr5
HOSTNAME: hello-tomcat-deployment-1638778051-rltr5

ServiceとDeploymentの削除

kubectl delete -f hello-tomcat.yml

一発でできた。

GKEクラスタの削除

gcloud container clusters delete hello-kube

は同じ内容をAzure Container Serviceで試す。