📝 BLOG.IK.AM

@making's memo
(🗃 Categories 🏷 Tags)

Kubernetesことはじめ (Azure Container Service編)

🗃 {Dev/CaaS/Kubernetes}

🏷 Azure Container Service 🏷 Kubernetes

🗓 Updated at 2017-06-06T17:57:26+09:00 by Toshiaki Maki  🗓 Created at 2017-06-05T13:02:03+09:00 by Toshiaki Maki  {✒️️ Edit  ⏰ History}


🚨 Warning: This content is old. Please don't trust too much.

前回でGKEでKubernetesを始めてみたが、今度は同じ内容をAzure Container Serviceで試す。

目次

事前準備

azコマンドをインストール。

curl -L https://aka.ms/InstallAzureCli | bash

バージョンはこちら

$ az --version
azure-cli (2.0.6)

acr (2.0.4)
acs (2.0.6)
appservice (0.1.6)
batch (2.0.4)
cdn (0.0.2)
cloud (2.0.2)
cognitiveservices (0.1.2)
command-modules-nspkg (2.0.0)
component (2.0.4)
configure (2.0.6)
core (2.0.6)
cosmosdb (0.1.6)
dla (0.0.6)
dls (0.0.6)
feedback (2.0.2)
find (0.2.2)
interactive (0.3.2)
iot (0.1.5)
keyvault (2.0.4)
lab (0.0.4)
monitor (0.0.4)
network (2.0.6)
nspkg (3.0.0)
profile (2.0.4)
rdbms (0.0.1)
redis (0.2.3)
resource (2.0.6)
role (2.0.4)
sf (1.0.1)
sql (2.0.3)
storage (2.0.6)
vm (2.0.6)

Python (Darwin) 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)]

Python location '/Users/makit/lib/azure-cli/bin/python'

メモ

自分はこの問題に当たった。。

k8sのバージョンはこちら。

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T20:41:07Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.3", GitCommit:"029c3a408176b55c30846f0faedf56aae5992e9b", GitTreeState:"clean", BuildDate:"2017-02-15T06:34:56Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

Azure ログインする。

az login

Kubernetesクラスタの作成

リソースグループの作成

az group create --name=kube-rg --location=japaneast

クラスタの作成。

とりあえずStandard_A1な小さめのワーカーインスタンスを1つだけ作る。(デフォルトはStandard_D2_v2が3インスタンス)

az acs create --orchestrator-type=kubernetes --resource-group=kube-rg --name=hello-kube --dns-prefix=makikube --generate-ssh-keys --agent-count=1 --agent-vm-size=Standard_A1

kube-rgリソースグループにこのくらいのリソースができている。

image

VMはmasterとworkerが一つずつできている。GKEの場合はmasterはGCP managedだった。

image

クラスタ情報を取得。

az acs kubernetes get-credentials --resource-group=kube-rg --name=hello-kube

これでACS上のKubernetesにアクセスできる。

$ kubectl cluster-info
Kubernetes master is running at https://makikube.japaneast.cloudapp.azure.com
Heapster is running at https://makikube.japaneast.cloudapp.azure.com/api/v1/proxy/namespaces/kube-system/services/heapster
KubeDNS is running at https://makikube.japaneast.cloudapp.azure.com/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at https://makikube.japaneast.cloudapp.azure.com/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Dashboardはkubectrl proxy経由でhttp://localhost:8001/uiでアクセスできる。

kubectl proxy

image

CLIだけでアプリケーションをデプロイする

まずはYAMLなしでデプロイ。

Deploymentの作成。

kubectl run hello-tomcat --image=making/hello-tomcat:v1 --port=8080

各種表示

$ kubectl get deployment -o wide
NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE       CONTAINER(S)   IMAGE(S)                 SELECTOR
hello-tomcat   1         1         1            1           21s       hello-tomcat   making/hello-tomcat:v1   run=hello-tomcat

$ kubectl get pod -o wide
NAME                            READY     STATUS    RESTARTS   AGE       IP           NODE
hello-tomcat-3358050292-01nnd   1/1       Running   0          55s       10.244.0.7   k8s-agent-37b09b98-0

$ kubectl get replicaset -o wide
NAME                      DESIRED   CURRENT   READY     AGE       CONTAINER(S)   IMAGE(S)                 SELECTOR
hello-tomcat-3358050292   1         1         1         1m        hello-tomcat   making/hello-tomcat:v1   pod-template-hash=3358050292,run=hello-tomcat

image

Deploymentの詳細表示

$ kubectl describe deployment hello-tomcat
Name:            hello-tomcat
Namespace:        default
CreationTimestamp:    Tue, 30 May 2017 13:17:07 +0900
Labels:            run=hello-tomcat
Annotations:        deployment.kubernetes.io/revision=1
Selector:        run=hello-tomcat
Replicas:        1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:        RollingUpdate
MinReadySeconds:    0
RollingUpdateStrategy:    1 max unavailable, 1 max surge
Pod Template:
  Labels:    run=hello-tomcat
  Containers:
   hello-tomcat:
    Image:        making/hello-tomcat:v1
    Port:        8080/TCP
    Environment:    <none>
    Mounts:        <none>
  Volumes:        <none>
Conditions:
  Type        Status    Reason
  ----        ------    ------
  Available     True    MinimumReplicasAvailable
OldReplicaSets:    <none>
NewReplicaSet:    <none>
Events:
  FirstSeen    LastSeen    Count    From            SubObjectPath    Type        Reason            Message
  ---------    --------    -----    ----            -------------    --------    ------            -------
  2m        2m        1    deployment-controller            Normal        ScalingReplicaSet    Scaled up replica set hello-tomcat-3358050292 to 1

Podの詳細表示

$ kubectl describe pod hello-tomcat-3358050292-01nnd
Name:        hello-tomcat-3358050292-01nnd
Namespace:    default
Node:        k8s-agent-37b09b98-0/10.240.0.4
Start Time:    Tue, 30 May 2017 13:17:08 +0900
Labels:        pod-template-hash=3358050292
        run=hello-tomcat
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"hello-tomcat-3358050292","uid":"d843c70f-44ee-11e7-bc98-000d3a50...
Status:        Running
IP:        10.244.0.7
Controllers:    ReplicaSet/hello-tomcat-3358050292
Containers:
  hello-tomcat:
    Container ID:    docker://7b0b2b23bb9e157f259d917ff24a0605d6fa376cf06b746918f2841ef740601d
    Image:        making/hello-tomcat:v1
    Image ID:        docker-pullable://making/[email protected]:1fe59a6acdce0964e1de2e41f3e5e736886fad619bb3cd9de2874cced60fd362
    Port:        8080/TCP
    State:        Running
      Started:        Tue, 30 May 2017 13:17:15 +0900
    Ready:        True
    Restart Count:    0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-bt6cd (ro)
Conditions:
  Type        Status
  Initialized     True
  Ready     True
  PodScheduled     True
Volumes:
  default-token-bt6cd:
    Type:    Secret (a volume populated by a Secret)
    SecretName:    default-token-bt6cd
    Optional:    false
QoS Class:    BestEffort
Node-Selectors:    <none>
Tolerations:    <none>
Events:
  FirstSeen    LastSeen    Count    From                SubObjectPath            Type        Reason        Message
  ---------    --------    -----    ----                -------------            --------    ------        -------
  3m        3m        1    default-scheduler                        Normal        Scheduled    Successfully assigned hello-tomcat-3358050292-01nnd to k8s-agent-37b09b98-0
  3m        3m        1    kubelet, k8s-agent-37b09b98-0    spec.containers{hello-tomcat}    Normal        Pulling        pulling image "making/hello-tomcat:v1"
  3m        3m        1    kubelet, k8s-agent-37b09b98-0    spec.containers{hello-tomcat}    Normal        Pulled        Successfully pulled image "making/hello-tomcat:v1"
  3m        3m        1    kubelet, k8s-agent-37b09b98-0    spec.containers{hello-tomcat}    Normal        Created        Created container with docker id 7b0b2b23bb9e; Security:[seccomp=unconfined]
  3m        3m        1    kubelet, k8s-agent-37b09b98-0    spec.containers{hello-tomcat}    Normal        Started        Started container with docker id 7b0b2b23bb9e

Replicasetの詳細表示

$ kubectl describe replicaset hello-tomcat-3358050292
Name:        hello-tomcat-3358050292
Namespace:    default
Selector:    pod-template-hash=3358050292,run=hello-tomcat
Labels:        pod-template-hash=3358050292
        run=hello-tomcat
Annotations:    deployment.kubernetes.io/desired-replicas=1
        deployment.kubernetes.io/max-replicas=2
        deployment.kubernetes.io/revision=1
Replicas:    1 current / 1 desired
Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:    pod-template-hash=3358050292
        run=hello-tomcat
  Containers:
   hello-tomcat:
    Image:        making/hello-tomcat:v1
    Port:        8080/TCP
    Environment:    <none>
    Mounts:        <none>
  Volumes:        <none>
Events:
  FirstSeen    LastSeen    Count    From            SubObjectPath    Type        Reason            Message
  ---------    --------    -----    ----            -------------    --------    ------            -------
  3m        3m        1    replicaset-controller            Normal        SuccessfulCreate    Created pod: hello-tomcat-3358050292-01nnd

コンテナの内部からcurlでWebアプリケーションにアクセス

$ kubectl exec -ti hello-tomcat-3358050292-01nnd /bin/bash

[email protected]:/usr/local/tomcat# curl -s http://localhost:8080/env | sort
CATALINA_HOME: /usr/local/tomcat
CA_CERTIFICATES_JAVA_VERSION: 20161107~bpo8+1
GPG_KEYS: 05AB33110949707C93A279E3D3EFE6B686867BA6 07E48665A34DCAFAE522E5E6266191C37C037D42 47309207D818FFD8DCD3F83F1931D684307A10A5 541FBE7D8F78B25E055DDEE13C370389288584E7 61B832AC2F1C5A90F0F9B00A1C506407564C17A3 713DA88BE50911535FE716F5208B0AB1D63011C7 79F7026C690BAA50B92CD8B66A3AD3F4F22C4FED 9BA44C2621385CB966EBA586F72C284D731FABEE A27677289986DB50844682F8ACB77FC2E86E29AC A9C5DF4D22E99998D9875A5110C01C5A2F6059E7 DCFD35E0BF8CA7344752DE8B6FB21E8933C60243 F3A04C595DB5B6A5F1ECA43E3B7BBB100D811BBE F7DA48BB64BCB84ECBA7EE6935CD23C10D498E23
HOME: /root
HOSTNAME: hello-tomcat-3358050292-01nnd
JAVA_DEBIAN_VERSION: 8u131-b11-1~bpo8+1
JAVA_HOME: /docker-java-home/jre
JAVA_VERSION: 8u131
KUBERNETES_PORT: tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP: tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP_ADDR: 10.0.0.1
KUBERNETES_PORT_443_TCP_PORT: 443
KUBERNETES_PORT_443_TCP_PROTO: tcp
KUBERNETES_SERVICE_HOST: 10.0.0.1
KUBERNETES_SERVICE_PORT: 443
KUBERNETES_SERVICE_PORT_HTTPS: 443
LANG: C.UTF-8
LD_LIBRARY_PATH: /usr/local/tomcat/native-jni-lib
OPENSSL_VERSION: 1.1.0e-2
PATH: /usr/local/tomcat/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD: /usr/local/tomcat
TOMCAT_ASC_URL: https://www.apache.org/dist/tomcat/tomcat-8/v8.5.15/bin/apache-tomcat-8.5.15.tar.gz.asc
TOMCAT_MAJOR: 8
TOMCAT_NATIVE_LIBDIR: /usr/local/tomcat/native-jni-lib
TOMCAT_TGZ_URL: https://www.apache.org/dyn/closer.cgi?action=download&filename=tomcat/tomcat-8/v8.5.15/bin/apache-tomcat-8.5.15.tar.gz
TOMCAT_VERSION: 8.5.15

Serviceの作成(外部トラフィックの許可)

外部トラフィックを受け付けるためのServiceを作成する。

kubectl expose deployment hello-tomcat --type=LoadBalancer

Serviceの情報を見る。

$ kubectl get service hello-tomcat -o wide
NAME           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE       SELECTOR
hello-tomcat   10.0.113.151   <pending>     8080:30699/TCP   39s       run=hello-tomcat

しばらくするとEXTERNAL-IPが割り振られる。

$ kubectl get service hello-tomcat -o wide
NAME           CLUSTER-IP     EXTERNAL-IP     PORT(S)          AGE       SELECTOR
hello-tomcat   10.0.113.151   xx.xx.xxx.xxx   8080:30699/TCP   3m        run=hello-tomcat

詳細表示。

$ kubectl describe service hello-tomcat
Name:            hello-tomcat
Namespace:        default
Labels:            run=hello-tomcat
Annotations:        <none>
Selector:        run=hello-tomcat
Type:            LoadBalancer
IP:            10.0.113.151
LoadBalancer Ingress:    xx.xx.xxx.xxx
Port:            <unset>    8080/TCP
NodePort:        <unset>    30699/TCP
Endpoints:        10.244.0.7:8080
Session Affinity:    None
Events:
  FirstSeen    LastSeen    Count    From            SubObjectPath    Type        Reason            Message
  ---------    --------    -----    ----            -------------    --------    ------            -------
  3m        3m        1    service-controller            Normal        CreatingLoadBalancer    Creating load balancer
  54s        54s        1    service-controller            Normal        CreatedLoadBalancer    Created load balancer

makikubeという名前のLoad balancerとPublic IP addressがリソースグループに追加されている。

image

Public IP addressの方を見ると、Load balancerとひもづいていることがわかる。

image

Load balancerを見るとWorker VMにひもづいていることがわかる。

image

ロードバランスのIP:8080でWebアプリケーションにアクセスできる。

$ curl -s xx.xx.xxx.xxx:8080/env | sort
CATALINA_HOME: /usr/local/tomcat
CA_CERTIFICATES_JAVA_VERSION: 20161107~bpo8+1
GPG_KEYS: 05AB33110949707C93A279E3D3EFE6B686867BA6 07E48665A34DCAFAE522E5E6266191C37C037D42 47309207D818FFD8DCD3F83F1931D684307A10A5 541FBE7D8F78B25E055DDEE13C370389288584E7 61B832AC2F1C5A90F0F9B00A1C506407564C17A3 713DA88BE50911535FE716F5208B0AB1D63011C7 79F7026C690BAA50B92CD8B66A3AD3F4F22C4FED 9BA44C2621385CB966EBA586F72C284D731FABEE A27677289986DB50844682F8ACB77FC2E86E29AC A9C5DF4D22E99998D9875A5110C01C5A2F6059E7 DCFD35E0BF8CA7344752DE8B6FB21E8933C60243 F3A04C595DB5B6A5F1ECA43E3B7BBB100D811BBE F7DA48BB64BCB84ECBA7EE6935CD23C10D498E23
HOME: /root
HOSTNAME: hello-tomcat-3358050292-01nnd
JAVA_DEBIAN_VERSION: 8u131-b11-1~bpo8+1
JAVA_HOME: /docker-java-home/jre
JAVA_VERSION: 8u131
KUBERNETES_PORT: tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP: tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP_ADDR: 10.0.0.1
KUBERNETES_PORT_443_TCP_PORT: 443
KUBERNETES_PORT_443_TCP_PROTO: tcp
KUBERNETES_SERVICE_HOST: 10.0.0.1
KUBERNETES_SERVICE_PORT: 443
KUBERNETES_SERVICE_PORT_HTTPS: 443
LANG: C.UTF-8
LD_LIBRARY_PATH: /usr/local/tomcat/native-jni-lib
OPENSSL_VERSION: 1.1.0e-2
PATH: /usr/local/tomcat/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD: /usr/local/tomcat
TOMCAT_ASC_URL: https://www.apache.org/dist/tomcat/tomcat-8/v8.5.15/bin/apache-tomcat-8.5.15.tar.gz.asc
TOMCAT_MAJOR: 8
TOMCAT_NATIVE_LIBDIR: /usr/local/tomcat/native-jni-lib
TOMCAT_TGZ_URL: https://www.apache.org/dyn/closer.cgi?action=download&filename=tomcat/tomcat-8/v8.5.15/bin/apache-tomcat-8.5.15.tar.gz
TOMCAT_VERSION: 8.5.15

スケールアウト

kubectl scale --replicas=2 deployment/hello-tomcat

各種表示

$ kubectl get deployment hello-tomcat -o wide
NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE       CONTAINER(S)   IMAGE(S)                 SELECTOR
hello-tomcat   2         2         2            2           3h        hello-tomcat   making/hello-tomcat:v1   run=hello-tomcat

$ kubectl get pod -l run=hello-tomcat -o wide
NAME                            READY     STATUS    RESTARTS   AGE       IP           NODE
hello-tomcat-3358050292-01nnd   1/1       Running   0          3h        10.244.0.7   k8s-agent-37b09b98-0
hello-tomcat-3358050292-4zgk7   1/1       Running   0          31s       10.244.0.8   k8s-agent-37b09b98-0

$ kubectl get replicaset -l run=hello-tomcat -o wide
NAME                      DESIRED   CURRENT   READY     AGE       CONTAINER(S)   IMAGE(S)                 SELECTOR
hello-tomcat-3358050292   2         2         2         3h        hello-tomcat   making/hello-tomcat:v1   pod-template-hash=3358050292,run=hello-tomcat

ServiceのEntpointsが2つに増えている。

$ kubectl describe service hello-tomcat
Name:            hello-tomcat
Namespace:        default
Labels:            run=hello-tomcat
Annotations:        <none>
Selector:        run=hello-tomcat
Type:            LoadBalancer
IP:            10.0.113.151
LoadBalancer Ingress:    xx.xx.xxx.xxx
Port:            <unset>    8080/TCP
NodePort:        <unset>    30699/TCP
Endpoints:        10.244.0.7:8080,10.244.0.8:8080
Session Affinity:    None
Events:            <none>

ロードバランスの確認。

$ for i in `seq 1 10`;do curl -s xx.xx.xxx.xxx:8080/env | grep HOSTNAME;done
HOSTNAME: hello-tomcat-3358050292-4zgk7
HOSTNAME: hello-tomcat-3358050292-4zgk7
HOSTNAME: hello-tomcat-3358050292-4zgk7
HOSTNAME: hello-tomcat-3358050292-4zgk7
HOSTNAME: hello-tomcat-3358050292-01nnd
HOSTNAME: hello-tomcat-3358050292-01nnd
HOSTNAME: hello-tomcat-3358050292-01nnd
HOSTNAME: hello-tomcat-3358050292-01nnd
HOSTNAME: hello-tomcat-3358050292-4zgk7
HOSTNAME: hello-tomcat-3358050292-01nnd

ServiceとDeploymentの削除

kubectl delete service hello-tomcat
kubectl delete deployment hello-tomcat

YAMLを使ったデプロイ

普通はYAMLを使ってデプロイする。

次のようなYAMLを作って

kind: Service
apiVersion: v1
metadata:
  name: hello-tomcat-service
spec:
  selector:
    run: hello-tomcat
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  type: LoadBalancer
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: hello-tomcat-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        run: hello-tomcat
    spec:
      containers:
      - name: hello-tomcat
        image: making/hello-tomcat
        ports:
        - containerPort: 8080

を作って

kubectl create -f hello-tomcat.yml

各種表示

$ kubectl get pod -o wide
NAME                                       READY     STATUS    RESTARTS   AGE       IP            NODE
hello-tomcat-deployment-3358050292-h7854   1/1       Running   0          3m        10.244.0.10   k8s-agent-37b09b98-0
hello-tomcat-deployment-3358050292-nr1l7   1/1       Running   0          3m        10.244.0.9    k8s-agent-37b09b98-0

$ kubectl get deployment hello-tomcat-deployment -o wide
NAME                      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE       CONTAINER(S)   IMAGE(S)                 SELECTOR
hello-tomcat-deployment   2         2         2            2           4m        hello-tomcat   making/hello-tomcat:v1   run=hello-tomcat

$ kubectl get service hello-tomcat-service -o wide
NAME                   CLUSTER-IP    EXTERNAL-IP     PORT(S)          AGE       SELECTOR
hello-tomcat-service   10.0.32.192   yy.yyy.yyy.yy   8080:30815/TCP   4m        run=hello-tomcat

ロードバランスの確認。

$ for i in `seq 1 10`;do curl -s yy.yyy.yyy.yy:8080/env | grep HOSTNAME;done
HOSTNAME: hello-tomcat-deployment-3358050292-nr1l7
HOSTNAME: hello-tomcat-deployment-3358050292-h7854
HOSTNAME: hello-tomcat-deployment-3358050292-nr1l7
HOSTNAME: hello-tomcat-deployment-3358050292-h7854
HOSTNAME: hello-tomcat-deployment-3358050292-nr1l7
HOSTNAME: hello-tomcat-deployment-3358050292-nr1l7
HOSTNAME: hello-tomcat-deployment-3358050292-nr1l7
HOSTNAME: hello-tomcat-deployment-3358050292-h7854
HOSTNAME: hello-tomcat-deployment-3358050292-h7854
HOSTNAME: hello-tomcat-deployment-3358050292-h7854

ServiceとDeploymentの削除

kubectl delete -f hello-tomcat.yml

一発でできた。

Kubernetesクラスタの削除

az acs delete --resource-group=kube-rg --name=hello-kube

image

Container Serviceのインスタンスは消えるけど、VMやLBは消えていないように見える。

全部消したい場合はリソースグループごと削除。

az group delete --name=kube-rg

次はStateful Applicationを試したい。