📝 BLOG.IK.AM

@making's memo
(🗃 Categories 🏷 Tags)

Kubernetesことはじめ (Minikube編)

🗃 {Dev/CaaS/Kubernetes}

🏷 Kubernetes 🏷 Minikube

🗓 Updated at 2017-06-06T17:59:12+09:00 by Toshiaki Maki  🗓 Created at 2017-06-05T13:01:07+09:00 by Toshiaki Maki  {✒️️ Edit  ⏰ History}


🚨 Warning: This content is old. Please don't trust too much.

Kubernetesも学ばないとね。個人的な当面のゴールは

  • Kuboを使ったKubernetesの運用ができるようになること。
  • Open Service Broker APIでKubernetes上でService Brokerを立ち上げて、Cloud Foundryのcf create-serviceで呼べるようになること。

そのために必要な知識をつけていきたい。できるだけ公式ドキュメントだけで頑張る。

目次

題材のアプリ

単純なJavaアプリケーションであるmaking/hello-tomcatを題材に使う。
Dockerを使ってローカルで動作確認できる。

docker run -p 8080:8080 making/hello-tomcat:v1

/envで環境変数を見ることができる。

$ curl localhost:8080/env
TOMCAT_TGZ_URL: https://www.apache.org/dyn/closer.cgi?action=download&filename=tomcat/tomcat-8/v8.5.15/bin/apache-tomcat-8.5.15.tar.gz
PATH: /usr/local/tomcat/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
JAVA_HOME: /docker-java-home/jre
CATALINA_HOME: /usr/local/tomcat
CA_CERTIFICATES_JAVA_VERSION: 20161107~bpo8+1
LANG: C.UTF-8
TOMCAT_MAJOR: 8
TOMCAT_VERSION: 8.5.15
HOSTNAME: ea7ef50f6ef4
JAVA_DEBIAN_VERSION: 8u131-b11-1~bpo8+1
LD_LIBRARY_PATH: /usr/local/tomcat/native-jni-lib
OPENSSL_VERSION: 1.1.0e-2
TOMCAT_NATIVE_LIBDIR: /usr/local/tomcat/native-jni-lib
TOMCAT_ASC_URL: https://www.apache.org/dist/tomcat/tomcat-8/v8.5.15/bin/apache-tomcat-8.5.15.tar.gz.asc
PWD: /usr/local/tomcat
JAVA_VERSION: 8u131
GPG_KEYS: 05AB33110949707C93A279E3D3EFE6B686867BA6 07E48665A34DCAFAE522E5E6266191C37C037D42 47309207D818FFD8DCD3F83F1931D684307A10A5 541FBE7D8F78B25E055DDEE13C370389288584E7 61B832AC2F1C5A90F0F9B00A1C506407564C17A3 713DA88BE50911535FE716F5208B0AB1D63011C7 79F7026C690BAA50B92CD8B66A3AD3F4F22C4FED 9BA44C2621385CB966EBA586F72C284D731FABEE A27677289986DB50844682F8ACB77FC2E86E29AC A9C5DF4D22E99998D9875A5110C01C5A2F6059E7 DCFD35E0BF8CA7344752DE8B6FB21E8933C60243 F3A04C595DB5B6A5F1ECA43E3B7BBB100D811BBE F7DA48BB64BCB84ECBA7EE6935CD23C10D498E23
HOME: /root

Minikubeのインストール

まずはMac上のMinikubeで試す。

各種バージョンはこちら。

$ minikube version
minikube version: v0.19.0
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T20:41:07Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-05-09T23:22:45Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}

Minikubeはbrewでインストール。

brew cask install minikube

xhyve driverを使う。

brew install docker-machine-driver-xhyve
sudo chown root:wheel $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
sudo chmod u+s $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve

start。

minikube start --vm-driver=xhyve

Dashboard表示

minikube dashboard

image

Cluster IP

$ minikube ip
192.168.64.2

CLIだけでアプリケーションをデプロイする

まずはYAMLなしでデプロイ。

Deploymentの作成。

kubectl run hello-tomcat --image=making/hello-tomcat:v1 --port=8080

各種表示

$ kubectl get deployment -o wide
NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE       CONTAINER(S)   IMAGE(S)                 SELECTOR
hello-tomcat   1         1         1            1           1h        hello-tomcat   making/hello-tomcat:v1   run=hello-tomcat

$ kubectl get pod -o wide
NAME                           READY     STATUS    RESTARTS   AGE       IP           NODE
hello-tomcat-906498948-k138h   1/1       Running   0          1h        172.17.0.4   minikube

$ kubectl get replicaset -o wide
NAME                     DESIRED   CURRENT   READY     AGE       CONTAINER(S)   IMAGE(S)                 SELECTOR
hello-tomcat-906498948   1         1         1         1h        hello-tomcat   making/hello-tomcat:v1   pod-template-hash=906498948,run=hello-tomcat

image

Deploymentの詳細表示

$ kubectl describe deployment hello-tomcat
Name:            hello-tomcat
Namespace:        default
CreationTimestamp:    Mon, 29 May 2017 23:03:20 +0900
Labels:            run=hello-tomcat
Annotations:        deployment.kubernetes.io/revision=1
Selector:        run=hello-tomcat
Replicas:        1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:        RollingUpdate
MinReadySeconds:    0
RollingUpdateStrategy:    1 max unavailable, 1 max surge
Pod Template:
  Labels:    run=hello-tomcat
  Containers:
   hello-tomcat:
    Image:        making/hello-tomcat:v1
    Port:        8080/TCP
    Environment:    <none>
    Mounts:        <none>
  Volumes:        <none>
Conditions:
  Type        Status    Reason
  ----        ------    ------
  Available     True    MinimumReplicasAvailable
OldReplicaSets:    <none>
NewReplicaSet:    hello-tomcat-906498948 (1/1 replicas created)
Events:
  FirstSeen    LastSeen    Count    From            SubObjectPath    Type        Reason            Message
  ---------    --------    -----    ----            -------------    --------    ------            -------
  1h        1h        1    deployment-controller            Normal        ScalingReplicaSet    Scaled up replica set hello-tomcat-906498948 to 1

Podの詳細表示

$ kubectl describe pod hello-tomcat-906498948-k138h
Name:        hello-tomcat-906498948-k138h
Namespace:    default
Node:        minikube/192.168.64.2
Start Time:    Mon, 29 May 2017 23:03:20 +0900
Labels:        pod-template-hash=906498948
        run=hello-tomcat
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"hello-tomcat-906498948","uid":"922a4949-4477-11e7-ba13-4217a125c...
Status:        Running
IP:        172.17.0.4
Controllers:    ReplicaSet/hello-tomcat-906498948
Containers:
  hello-tomcat:
    Container ID:    docker://78f65eb922d83555be8311f924e6f43ce6d4a8783c267e1376d2ec2fc032bd3e
    Image:        making/hello-tomcat:v1
    Image ID:        docker://sha256:a59c0230b53e29078b0725c35c8b64f3d276b31a59a0b8cd5f7226eec7fc2d7a
    Port:        8080/TCP
    State:        Running
      Started:        Mon, 29 May 2017 23:03:23 +0900
    Ready:        True
    Restart Count:    0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wtt9g (ro)
Conditions:
  Type        Status
  Initialized     True
  Ready     True
  PodScheduled     True
Volumes:
  default-token-wtt9g:
    Type:    Secret (a volume populated by a Secret)
    SecretName:    default-token-wtt9g
    Optional:    false
QoS Class:    BestEffort
Node-Selectors:    <none>
Tolerations:    <none>
Events:
  FirstSeen    LastSeen    Count    From            SubObjectPath            Type        Reason        Message
  ---------    --------    -----    ----            -------------            --------    ------        -------
  1h        1h        1    default-scheduler                    Normal        Scheduled    Successfully assigned hello-tomcat-906498948-k138h to minikube
  1h        1h        1    kubelet, minikube    spec.containers{hello-tomcat}    Normal        Pulling        pulling image "making/hello-tomcat:v1"
  1h        1h        1    kubelet, minikube    spec.containers{hello-tomcat}    Normal        Pulled        Successfully pulled image "making/hello-tomcat:v1"
  1h        1h        1    kubelet, minikube    spec.containers{hello-tomcat}    Normal        Created        Created container with id 78f65eb922d83555be8311f924e6f43ce6d4a8783c267e1376d2ec2fc032bd3e
  1h        1h        1    kubelet, minikube    spec.containers{hello-tomcat}    Normal        Started        Started container with id 78f65eb922d83555be8311f924e6f43ce6d4a8783c267e1376d2ec2fc032bd3e

Replicasetの詳細表示

$ kubectl describe replicaset hello-tomcat-906498948
Name:        hello-tomcat-906498948
Namespace:    default
Selector:    pod-template-hash=906498948,run=hello-tomcat
Labels:        pod-template-hash=906498948
        run=hello-tomcat
Annotations:    deployment.kubernetes.io/desired-replicas=1
        deployment.kubernetes.io/max-replicas=2
        deployment.kubernetes.io/revision=1
Replicas:    1 current / 1 desired
Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:    pod-template-hash=906498948
        run=hello-tomcat
  Containers:
   hello-tomcat:
    Image:        making/hello-tomcat:v1
    Port:        8080/TCP
    Environment:    <none>
    Mounts:        <none>
  Volumes:        <none>
Events:
  FirstSeen    LastSeen    Count    From            SubObjectPath    Type        Reason            Message
  ---------    --------    -----    ----            -------------    --------    ------            -------
  1h        1h        1    replicaset-controller            Normal        SuccessfulCreate    Created pod: hello-tomcat-906498948-k138h

コンテナの内部からcurlでWebアプリケーションにアクセス

$ kubectl exec -ti hello-tomcat-906498948-k138h /bin/bash

root@hello-tomcat-906498948-k138h:/usr/local/tomcat# curl -s http://localhost:8080/env | sort
CATALINA_HOME: /usr/local/tomcat
CA_CERTIFICATES_JAVA_VERSION: 20161107~bpo8+1
GPG_KEYS: 05AB33110949707C93A279E3D3EFE6B686867BA6 07E48665A34DCAFAE522E5E6266191C37C037D42 47309207D818FFD8DCD3F83F1931D684307A10A5 541FBE7D8F78B25E055DDEE13C370389288584E7 61B832AC2F1C5A90F0F9B00A1C506407564C17A3 713DA88BE50911535FE716F5208B0AB1D63011C7 79F7026C690BAA50B92CD8B66A3AD3F4F22C4FED 9BA44C2621385CB966EBA586F72C284D731FABEE A27677289986DB50844682F8ACB77FC2E86E29AC A9C5DF4D22E99998D9875A5110C01C5A2F6059E7 DCFD35E0BF8CA7344752DE8B6FB21E8933C60243 F3A04C595DB5B6A5F1ECA43E3B7BBB100D811BBE F7DA48BB64BCB84ECBA7EE6935CD23C10D498E23
HOME: /root
HOSTNAME: hello-tomcat-906498948-k138h
JAVA_DEBIAN_VERSION: 8u131-b11-1~bpo8+1
JAVA_HOME: /docker-java-home/jre
JAVA_VERSION: 8u131
KUBERNETES_PORT: tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP: tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP_ADDR: 10.0.0.1
KUBERNETES_PORT_443_TCP_PORT: 443
KUBERNETES_PORT_443_TCP_PROTO: tcp
KUBERNETES_SERVICE_HOST: 10.0.0.1
KUBERNETES_SERVICE_PORT: 443
KUBERNETES_SERVICE_PORT_HTTPS: 443
LANG: C.UTF-8
LD_LIBRARY_PATH: /usr/local/tomcat/native-jni-lib
OPENSSL_VERSION: 1.1.0e-2
PATH: /usr/local/tomcat/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD: /usr/local/tomcat
TOMCAT_ASC_URL: https://www.apache.org/dist/tomcat/tomcat-8/v8.5.15/bin/apache-tomcat-8.5.15.tar.gz.asc
TOMCAT_MAJOR: 8
TOMCAT_NATIVE_LIBDIR: /usr/local/tomcat/native-jni-lib
TOMCAT_TGZ_URL: https://www.apache.org/dyn/closer.cgi?action=download&filename=tomcat/tomcat-8/v8.5.15/bin/apache-tomcat-8.5.15.tar.gz
TOMCAT_VERSION: 8.5.15

Serviceの作成(外部トラフィックの許可)

外部トラフィックを受け付けるためのServiceを作成する。

kubectl expose deployment hello-tomcat --type=LoadBalancer

Serviceの情報を見る。

$ kubectl get service hello-tomcat -o wide
NAME           CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE       SELECTOR
hello-tomcat   10.0.0.72    <pending>     8080:32387/TCP   1h        run=hello-tomcat

Minikubeだと--type=LoadBalancerにしても意味はなく、EXTERNAL-IP<pending>のまま。--type=NodePortと同じ。

詳細表示。

$ kubectl describe service hello-tomcat
Name:            hello-tomcat
Namespace:        default
Labels:            run=hello-tomcat
Annotations:        <none>
Selector:        run=hello-tomcat
Type:            LoadBalancer
IP:            10.0.0.72
Port:            <unset>    8080/TCP
NodePort:        <unset>    32387/TCP
Endpoints:        172.17.0.4:8080
Session Affinity:    None
Events:            <none>

minikube ipで出力されるIPのNodePortのPortでWebアプリケーションにアクセスできる。

$ curl -s `minikube ip`:32387/env | sort
CATALINA_HOME: /usr/local/tomcat
CA_CERTIFICATES_JAVA_VERSION: 20161107~bpo8+1
GPG_KEYS: 05AB33110949707C93A279E3D3EFE6B686867BA6 07E48665A34DCAFAE522E5E6266191C37C037D42 47309207D818FFD8DCD3F83F1931D684307A10A5 541FBE7D8F78B25E055DDEE13C370389288584E7 61B832AC2F1C5A90F0F9B00A1C506407564C17A3 713DA88BE50911535FE716F5208B0AB1D63011C7 79F7026C690BAA50B92CD8B66A3AD3F4F22C4FED 9BA44C2621385CB966EBA586F72C284D731FABEE A27677289986DB50844682F8ACB77FC2E86E29AC A9C5DF4D22E99998D9875A5110C01C5A2F6059E7 DCFD35E0BF8CA7344752DE8B6FB21E8933C60243 F3A04C595DB5B6A5F1ECA43E3B7BBB100D811BBE F7DA48BB64BCB84ECBA7EE6935CD23C10D498E23
HOME: /root
HOSTNAME: hello-tomcat-906498948-k138h
JAVA_DEBIAN_VERSION: 8u131-b11-1~bpo8+1
JAVA_HOME: /docker-java-home/jre
JAVA_VERSION: 8u131
KUBERNETES_PORT: tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP: tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP_ADDR: 10.0.0.1
KUBERNETES_PORT_443_TCP_PORT: 443
KUBERNETES_PORT_443_TCP_PROTO: tcp
KUBERNETES_SERVICE_HOST: 10.0.0.1
KUBERNETES_SERVICE_PORT: 443
KUBERNETES_SERVICE_PORT_HTTPS: 443
LANG: C.UTF-8
LD_LIBRARY_PATH: /usr/local/tomcat/native-jni-lib
OPENSSL_VERSION: 1.1.0e-2
PATH: /usr/local/tomcat/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD: /usr/local/tomcat
TOMCAT_ASC_URL: https://www.apache.org/dist/tomcat/tomcat-8/v8.5.15/bin/apache-tomcat-8.5.15.tar.gz.asc
TOMCAT_MAJOR: 8
TOMCAT_NATIVE_LIBDIR: /usr/local/tomcat/native-jni-lib
TOMCAT_TGZ_URL: https://www.apache.org/dyn/closer.cgi?action=download&filename=tomcat/tomcat-8/v8.5.15/bin/apache-tomcat-8.5.15.tar.gz
TOMCAT_VERSION: 8.5.15

スケールアウト

kubectl scale --replicas=3 deployment/hello-tomcat

各種表示

$ kubectl get deployment hello-tomcat -o wide
NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE       CONTAINER(S)   IMAGE(S)                 SELECTOR
hello-tomcat   3         3         3            3           1h        hello-tomcat   making/hello-tomcat:v1   run=hello-tomcat

$ kubectl get pod -l run=hello-tomcat -o wide
NAME                           READY     STATUS    RESTARTS   AGE       IP           NODE
hello-tomcat-906498948-bqfxk   1/1       Running   0          1h        172.17.0.5   minikube
hello-tomcat-906498948-jkppv   1/1       Running   0          1h        172.17.0.6   minikube
hello-tomcat-906498948-k138h   1/1       Running   0          1h        172.17.0.4   minikube

$ kubectl get replicaset -l run=hello-tomcat -o wide
NAME                     DESIRED   CURRENT   READY     AGE       CONTAINER(S)   IMAGE(S)                 SELECTOR
hello-tomcat-906498948   3         3         3         1h        hello-tomcat   making/hello-tomcat:v1   pod-template-hash=906498948,run=hello-tomcat

ServiceのEntpointsが3つに増えている。

$ kubectl describe service hello-tomcat
Name:            hello-tomcat
Namespace:        default
Labels:            run=hello-tomcat
Annotations:        <none>
Selector:        run=hello-tomcat
Type:            LoadBalancer
IP:            10.0.0.72
Port:            <unset>    8080/TCP
NodePort:        <unset>    32387/TCP
Endpoints:        172.17.0.4:8080,172.17.0.5:8080,172.17.0.6:8080
Session Affinity:    None
Events:            <none>

ロードバランスの確認。

$ for i in `seq 1 10`;do curl -s `minikube ip`:32387/env | grep HOSTNAME;done
HOSTNAME: hello-tomcat-906498948-jkppv
HOSTNAME: hello-tomcat-906498948-jkppv
HOSTNAME: hello-tomcat-906498948-bqfxk
HOSTNAME: hello-tomcat-906498948-k138h
HOSTNAME: hello-tomcat-906498948-jkppv
HOSTNAME: hello-tomcat-906498948-k138h
HOSTNAME: hello-tomcat-906498948-k138h
HOSTNAME: hello-tomcat-906498948-k138h
HOSTNAME: hello-tomcat-906498948-bqfxk
HOSTNAME: hello-tomcat-906498948-k138h

ServiceとDeploymentの削除

kubectl delete service hello-tomcat
kubectl delete deployment hello-tomcat

YAMLを使ったデプロイ

普通はYAMLを使ってデプロイする。

次のようなYAMLを作って

kind: Service
apiVersion: v1
metadata:
  name: hello-tomcat-service
spec:
  selector:
    run: hello-tomcat
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  type: LoadBalancer
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: hello-tomcat-deployment
spec:
  replicas: 3
  template:
    metadata:
      labels:
        run: hello-tomcat
    spec:
      containers:
      - name: hello-tomcat
        image: making/hello-tomcat:v1
        ports:
        - containerPort: 8080

を作って

kubectl create -f hello-tomcat.yml

各種表示

$ kubectl get pod -o wide
NAME                                      READY     STATUS    RESTARTS   AGE       IP           NODE
hello-tomcat-deployment-906498948-5qnsn   1/1       Running   0          1h        172.17.0.5   minikube
hello-tomcat-deployment-906498948-r4836   1/1       Running   0          1h        172.17.0.4   minikube
hello-tomcat-deployment-906498948-wtk16   1/1       Running   0          1h        172.17.0.6   minikube

$ kubectl get deployment hello-tomcat-deployment -o wide
NAME                      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE       CONTAINER(S)   IMAGE(S)                 SELECTOR
hello-tomcat-deployment   3         3         3            3           1h        hello-tomcat   making/hello-tomcat:v1   run=hello-tomcat

$ kubectl get service hello-tomcat-service -o wide
NAME                   CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE       SELECTOR
hello-tomcat-service   10.0.0.67    <pending>     8080:32030/TCP   1h        run=hello-tomcat

ロードバランスの確認。

$ for i in `seq 1 10`;do curl -s `minikube ip`:32030/env | grep HOSTNAME;done
HOSTNAME: hello-tomcat-deployment-906498948-r4836
HOSTNAME: hello-tomcat-deployment-906498948-wtk16
HOSTNAME: hello-tomcat-deployment-906498948-r4836
HOSTNAME: hello-tomcat-deployment-906498948-r4836
HOSTNAME: hello-tomcat-deployment-906498948-5qnsn
HOSTNAME: hello-tomcat-deployment-906498948-r4836
HOSTNAME: hello-tomcat-deployment-906498948-wtk16
HOSTNAME: hello-tomcat-deployment-906498948-r4836
HOSTNAME: hello-tomcat-deployment-906498948-wtk16
HOSTNAME: hello-tomcat-deployment-906498948-5qnsn

ServiceとDeploymentの削除

kubectl delete -f hello-tomcat.yml

一発でできた。

は同じ内容をGoogle Container Engineで試す。