Cloud Foundry(v243)をDiegoオンリーで小さめな構成でAWSにデプロイ

OSSのCloud FoundryをAWSにインストールする方法。

前回、BOSH-LiteでCloud Foundryをインストールしましたが、基本的にはこれと同じマニフェストファイルを使用します。Diego Onlyです。

使用するソフトウェアのバージョン

BOSH ... 257.14
BOSH AWS CPI ... 60
BOSH CLI ... 1.3215.0
BOSH INIT CLI ... 0.0.81-775439c-2015-12-09T00:36:03Z
cf ... 243
cflinuxfs2-rootfs ... 1.29.0
diego ... 0.1485.0
etcd ... 67
garden-linux ... 0.342.0
stemcell ... 3262.14

全体像

Cloud FoundryのインストールにはBOSHという分散ソフトウェアのプロビジョニングツールを使用します。BOSHでソフトウェアをインストールするには最初にBOSH Directorという司令塔となるVMを立ち上げて、Director経由で各種VMをプロビジョニングすることになります。

BOSH

流れとしては

VPCをセットアップ(NAT含む)
BOSH Directorのセットアップ
Cloud Foundryのインストール

となります。図で表現すると次のような構成になります。青がBOSH (Direcotr)、赤がAWS Managed、緑がBOSH Managed VM(ここでは所謂Cloud FoundryのRuntime)です。

ProductionでHA Proxyでは非推奨なので、後ほどHA Proxyの代わりにELBを使う方法を紹介します。

本当はMulti-AZ構成も取れるけれど、今回はできるだけ小さい構成にします。

VPCのセットアップ

まずはVPCのセットアップからです。どこかにCloud Formationのjsonが転がっていると思いますが、今回は構成を理解するためにも、地道にWizardで作成します。先にNATゲートウェイ用のEIPを払い出しておきます。

"Start VPC Wizard"をクリック。

"VPC with Public and Private Subnets"を選んで"Select"をクリック。

サブネットの設定。スクリーンキャプチャの通り入力してください。EIPは先ほど作成したものです。

"Creaet VPC"をクリック。

"OK"をクリック。

VPCができました。

publicとprivateのSubnet IDをメモしておいてください。

次にKey Pairs。

boshという名前のkeyparを作成してください。bosh.pemはあとで使うのでダウンロードしておいてください。

次にBOSH Directorのセキュリティグループを作成。次のスクリーンキャプチャの通り。グレーアウトしている箇所はBOSH Directorにアクセスする端末のIPアドレス。とりあえず全開放でいいなら0.0.0.0/0を設定しておいてあとで直すってのでもOK。ALL TCPとALL UDPはboshセキュリティグループがSourceのものにしておいてください。

次にBOSH Directorに設定するEIPを作成します。

いよいよBOSH Directorをインストールするためのマニフェストを作成します。

mkdir ~/my-bosh
cd ~/my-bosh
cp <pemをダウンロードしたフォルダ>/bosh.pem ~/

マニフェストはドキュメントからコピーして穴埋め。~/my-bosh/bosh-aws.ymlというファイル名で。

---
name: bosh

releases:
- name: bosh
  url: https://bosh.io/d/github.com/cloudfoundry/bosh?v=257.14
  sha1: b41821dccee78ad7bd44466cd0160920c183787a
- name: bosh-aws-cpi
  url: https://bosh.io/d/github.com/cloudfoundry-incubator/bosh-aws-cpi-release?v=60
  sha1: 8e40a9ff892204007889037f094a1b0d23777058

resource_pools:
- name: vms
  network: private
  stemcell:
    url: https://bosh.io/d/stemcells/bosh-aws-xen-hvm-ubuntu-trusty-go_agent?v=3263.2
    sha1: 36a44a0f8eb866d7cf51ae208b6a677ec0bb6888
  cloud_properties:
    instance_type: t2.medium # <- 推奨はm3.xlargeだけど今回は小さいのを使う
    ephemeral_disk: {size: 25_000, type: gp2}
    availability_zone: ap-northeast-1a # <--- Replace with Availability Zone (済)

disk_pools:
- name: disks
  disk_size: 20_000
  cloud_properties: {type: gp2}

networks:
- name: private
  type: manual
  subnets:
  - range: 10.0.0.0/24
    gateway: 10.0.0.1
    dns: [10.0.0.2]
    cloud_properties: {subnet: SUBNET-ID} # <--- Replace with Public Subnet ID
- name: public
  type: vip

jobs:
- name: bosh
  instances: 1

  templates:
  - {name: nats, release: bosh}
  - {name: postgres, release: bosh}
  - {name: blobstore, release: bosh}
  - {name: director, release: bosh}
  - {name: health_monitor, release: bosh}
  - {name: registry, release: bosh}
  - {name: aws_cpi, release: bosh-aws-cpi}

  resource_pool: vms
  persistent_disk_pool: disks

  networks:
  - name: private
    static_ips: [10.0.0.6]
    default: [dns, gateway]
  - name: public
    static_ips: [ELASTIC-IP] # <--- Replace with Elastic IP

  properties:
    nats:
      address: 127.0.0.1
      user: nats
      password: nats-password

    postgres: &db
      listen_address: 127.0.0.1
      host: 127.0.0.1
      user: postgres
      password: postgres-password
      database: bosh
      adapter: postgres

    registry:
      address: 10.0.0.6
      host: 10.0.0.6
      db: *db
      http: {user: admin, password: admin, port: 25777}
      username: admin
      password: admin
      port: 25777

    blobstore:
      address: 10.0.0.6
      port: 25250
      provider: dav
      director: {user: director, password: director-password}
      agent: {user: agent, password: agent-password}

    director:
      address: 127.0.0.1
      name: my-bosh
      db: *db
      cpi_job: aws_cpi
      max_threads: 10
      user_management:
        provider: local
        local:
          users:
          - {name: admin, password: admin}
          - {name: hm, password: hm-password}

    hm:
      director_account: {user: hm, password: hm-password}
      resurrector_enabled: true

    aws: &aws
      access_key_id: ACCESS-KEY-ID # <--- Replace with AWS Access Key ID
      secret_access_key: SECRET-ACCESS-KEY # <--- Replace with AWS Secret Key
      default_key_name: bosh
      default_security_groups: [bosh]
      region: ap-northeast-1  # <--- Replace with Region (済)

    agent: {mbus: "nats://nats:nats-password@10.0.0.6:4222"}

    ntp: &ntp [0.pool.ntp.org, 1.pool.ntp.org]

cloud_provider:
  template: {name: aws_cpi, release: bosh-aws-cpi}

  ssh_tunnel:
    host: ELASTIC-IP # <--- Replace with your Elastic IP address
    port: 22
    user: vcap
    private_key: ./bosh.pem # Path relative to this manifest file

  mbus: "https://mbus:mbus-password@ELASTIC-IP:6868" # <--- Replace with Elastic IP

  properties:
    aws: *aws
    agent: {mbus: "https://mbus:mbus-password@0.0.0.0:6868"}
    blobstore: {provider: local, path: /var/vcap/micro_bosh/data/cache}
    ntp: *ntp

Replace ...の部分を自分の環境に合わせて埋めてください

SUBNET-ID ... cf_publicのSubnet ID
ELASTIC-IP ... 払い出したEIP
ACCESS-KEY-ID ... bosh keypairのAccesse Key ID
SECRET-ACCESS-KEY ... bosh keypairのSecret Access Key

BOSH Directorをインストールするためのbosh-initコマンドをこちらからダウンロードしてパスの通った場所に配置。

cd ~/my-bosh
bosh-init deploy bosh-aws.yml

30分ほどでインストールできます。ちなみにMac OS El Captanの環境でやるとNokogiriのインストールがうまくいかず途中でこけました。自分は諦めてUbuntu環境で再トライしたらすんなり行きました。セキュリティグループに注意。タイムアウト系のエラーが出たら大体セキュリティグループの問題です。サブネットIDやInboundの設定を確認してください。

EC2のインスタンス一覧にbosh/0という名前のVMが立っているはずです。

Cloud Foundryのデプロイ

BOSH Directorがデプロイできたら次はBOSH CLIでDirectorにログインしてCloud Foundryのデプロイです。

$ gem install bosh_cli
$ bosh target <BOSH DirectorのEIP>
RSA 1024 bit CA certificates are loaded due to old openssl compatibility
Target set to 'my-bosh'
Your username: admin
Enter password: admin
Logged in as 'admin'

Public SubnetにいるVMに対するセキュリティグループをcf-publicという名前で作ります。Inbound Rulesは次の通り。

4443はWebSocket（のちに使うELB対策）用です。ログやメトリクスを取得するときに使います。

ロードバランサのHA_Proxy用にElastic IPをもう一つ用意します。

このIPが外からCloud Foundryにアクセスするための入り口になるので、ドメインに登録します。

今回は

システム用ドメイン名をsytem.awscf.ik.am
アプリ用ドメイン名をapps.awscf.ik.am

にします。一つにまとめてもいいですが、分けるのがオススメです。ドメイン名を持っていない人は<HA_PROXYのEIP>.xip.ioで代用してください。

HA_PROXYのEIPを*.awscf.ik.amのAレコードで登録します。Route53を使ってもいいですが、次の例ではCloudFlareを使っています。

最後の難関、Cloud Foundryのマニフェストの用意です。幸いDiego OnlyのAWS用マニフェストがcf-releaseプロジェクトで用意されているので、それを使います。重要なコンポーネントを分割した最小構成らしいです。

少しだけカスタマイズして貼り付けておきます。AZとSecurity Groupは上記の内容で設定済みです。検証用のためinstance_typeをだいぶ貧弱にしてあります。 Cell VMはメモリ16GB以上推奨です。

aws-cf.yamlにいかの内容を貼り付けてください。

# The following line helps maintain current documentation at http://docs.cloudfoundry.org.
# code_snippet cf-minimal-aws start
---
name: cf
director_uuid: <%= `bosh status --uuid` %>

releases:
- {name: cf, version: latest}
- {name: diego, version: latest}
- {name: etcd, version: latest}
- {name: garden-linux, version: latest}
- {name: cflinuxfs2-rootfs , version: latest}

networks:
- name: cf_private
  type: manual
  subnets:
  - range: 10.0.16.0/24
    gateway: 10.0.16.1
    dns: [10.0.0.2]
    reserved: ["10.0.16.2 - 10.0.16.3"]
    static: ["10.0.16.100 - 10.0.16.105"]
    cloud_properties:
      subnet: REPLACE_WITH_PRIVATE_SUBNET_ID

- name: cf_public
  type: manual
  subnets:
  - range: 10.0.0.0/24
    gateway: 10.0.0.1
    dns: [10.0.0.2]
    reserved: ["10.0.0.2 - 10.0.0.10"]
    cloud_properties:
      subnet: REPLACE_WITH_PUBLIC_SUBNET_ID
      security_groups:
        - cf-public
        - bosh

- name: elastic
  type: vip
  cloud_properties: {}

resource_pools:
- name: small_z1
  network: cf_private
  stemcell:
    name: bosh-aws-xen-hvm-ubuntu-trusty-go_agent
    version: latest
  cloud_properties:
    availability_zone: ap-northeast-1a
    instance_type: t2.medium # 元の設定はc3.large
- name: diego_z1
  network: cf_private
  stemcell:
    name: bosh-aws-xen-hvm-ubuntu-trusty-go_agent
    version: latest
  cloud_properties:
    availability_zone: ap-northeast-1a
    ephemeral_disk:
      size: 40000
    instance_type: m3.xlarge # 元の設定はm3.2xlarge
- name: diego_cell_z1
  network: cf_private
  stemcell:
    name: bosh-aws-xen-hvm-ubuntu-trusty-go_agent
    version: latest
  cloud_properties:
    availability_zone: ap-northeast-1a
    ephemeral_disk:
      size: 80000
      type: io1
      iops: 1200
    instance_type: m3.xlarge # 元の設定はm3.2xlarge

compilation:
  workers: 6
  network: cf_private
  reuse_compilation_vms: true
  cloud_properties:
    availability_zone: ap-northeast-1a
    instance_type: c3.large

update:
  canaries: 1
  max_in_flight: 1
  serial: false
  canary_watch_time: 30000-600000
  update_watch_time: 5000-600000

jobs:
- name: nats_z1
  instances: 1
  resource_pool: small_z1
  templates:
  - {name: nats, release: cf}
  - {name: nats_stream_forwarder, release: cf}
  - {name: metron_agent, release: cf}
  networks:
  - name: cf_private
    static_ips: [10.0.16.103]

- name: etcd_z1
  instances: 1
  resource_pool: small_z1
  persistent_disk: 102400
  templates:
  - {name: etcd, release: cf}
  - {name: etcd_metrics_server, release: cf}
  - {name: metron_agent, release: cf}
  networks:
  - name: cf_private
    static_ips: [10.0.16.104]
  properties:
    etcd_metrics_server:
      nats:
        machines: [10.0.16.103]
        password: PASSWORD
        username: nats

- name: consul_z1
  instances: 1
  persistent_disk: 1024
  resource_pool: small_z1
  templates:
  - {name: metron_agent, release: cf}
  - {name: consul_agent, release: cf}
  networks:
  - name: cf_private
    static_ips: [10.0.16.105]
  properties:
    consul:
      agent:
        mode: server

- name: diego_cell_z1
  instances: 1
  resource_pool: diego_cell_z1
  templates:
  - name: consul_agent
    release: cf
  - name: rep
    release: diego
  - name: garden
    release: garden-linux
  - name: cflinuxfs2-rootfs-setup
    release: cflinuxfs2-rootfs
  - name: metron_agent
    release: cf
  networks:
  - name: cf_private
  update:
    serial: true
    max_in_flight: 1
  properties:
    metron_agent:
      zone: z1
    diego:
      rep:
        zone: z1

- name: diego_brain_z1
  instances: 1
  resource_pool: diego_z1
  templates:
  - name: consul_agent
    release: cf
  - name: etcd
    release: etcd
  - name: bbs
    release: diego
  - name: auctioneer
    release: diego
  - name: stager
    release: cf
  - name: nsync
    release: cf
  - name: tps
    release: cf
  - name: cc_uploader
    release: cf
  - name: file_server
    release: diego
  - name: route_emitter
    release: diego
  - name: metron_agent
    release: cf
  persistent_disk: 20480
  networks:
  - name: cf_private
  update:
    serial: true
    max_in_flight: 1
  properties:
    consul:
      agent:
        services:
          etcd: {}
    metron_agent:
      zone: z1

- name: blobstore_z1
  instances: 1
  persistent_disk: 102400
  resource_pool: small_z1
  templates:
  - {name: blobstore, release: cf}
  - {name: metron_agent, release: cf}
  - {name: route_registrar, release: cf}
  - {name: consul_agent, release: cf}
  networks:
  - name: cf_private
  properties:
    consul:
      agent:
        services:
          blobstore: {}
    route_registrar:
      routes:
      - name: blobstore
        port: 8080
        registration_interval: 20s
        tags:
          component: blobstore
        uris:
        - "blobstore.REPLACE_WITH_SYSTEM_DOMAIN"

- name: postgres_z1
  instances: 1
  persistent_disk: 1024
  resource_pool: small_z1
  templates:
  - {name: postgres, release: cf}
  - {name: metron_agent, release: cf}
  networks:
  - name: cf_private
    static_ips: [10.0.16.101]
  update:
    serial: true

- name: api_z1
  instances: 1
  resource_pool: small_z1
  templates:
  - {name: cloud_controller_ng, release: cf}
  - {name: cloud_controller_worker, release: cf}
  - {name: cloud_controller_clock, release: cf}
  - {name: metron_agent, release: cf}
  - {name: route_registrar, release: cf}
  - {name: consul_agent, release: cf}
  - {name: go-buildpack, release: cf}
  - {name: binary-buildpack, release: cf}
  - {name: nodejs-buildpack, release: cf}
  - {name: ruby-buildpack, release: cf}
  - {name: java-buildpack, release: cf}
  - {name: php-buildpack, release: cf}
  - {name: python-buildpack, release: cf}
  - {name: staticfile-buildpack, release: cf}
  networks:
  - name: cf_private
  properties:
    consul:
      agent:
        services:
          cloud_controller_ng: {}
    route_registrar:
      routes:
      - name: api
        registration_interval: 20s
        port: 9022
        uris:
        - "api.REPLACE_WITH_SYSTEM_DOMAIN"

- name: ha_proxy_z1
  instances: 1
  resource_pool: small_z1
  templates:
  - {name: haproxy, release: cf}
  - {name: consul_agent, release: cf}
  - {name: metron_agent, release: cf}
  networks:
  - name: elastic
    static_ips: [REPLACE_WITH_ELASTIC_IP]
  - name: cf_public
    default: [gateway, dns]
  properties:
    ha_proxy:
      ssl_pem: |
        REPLACE_WITH_SSL_CERT_AND_KEY
    router:
      servers:
        - 10.0.16.102

- name: doppler_z1
  instances: 1
  resource_pool: small_z1
  templates:
  - {name: doppler, release: cf}
  - {name: metron_agent, release: cf}
  - {name: syslog_drain_binder, release: cf}
  networks:
  - name: cf_private
  properties:
    doppler: {zone: z1}
    doppler_endpoint:
      shared_secret: PASSWORD

- name: loggregator_trafficcontroller_z1
  instances: 1
  resource_pool: small_z1
  templates:
  - {name: loggregator_trafficcontroller, release: cf}
  - {name: metron_agent, release: cf}
  - {name: route_registrar, release: cf}
  networks:
  - name: cf_private
  properties:
    traffic_controller: {zone: z1}
    route_registrar:
      routes:
      - name: doppler
        registration_interval: 20s
        port: 8081
        uris:
        - "doppler.REPLACE_WITH_SYSTEM_DOMAIN"
      - name: loggregator
        registration_interval: 20s
        port: 8080
        uris:
          - "loggregator.REPLACE_WITH_SYSTEM_DOMAIN"

- name: uaa_z1
  instances: 1
  resource_pool: small_z1
  templates:
  - {name: uaa, release: cf}
  - {name: metron_agent, release: cf}
  - {name: route_registrar, release: cf}
  networks:
  - name: cf_private
  properties:
    login:
      catalina_opts: -Xmx768m -XX:MaxPermSize=256m
    route_registrar:
      routes:
      - name: uaa
        registration_interval: 20s
        port: 8080
        uris:
        - "uaa.REPLACE_WITH_SYSTEM_DOMAIN"
        - "*.uaa.REPLACE_WITH_SYSTEM_DOMAIN"
        - "login.REPLACE_WITH_SYSTEM_DOMAIN"
        - "*.login.REPLACE_WITH_SYSTEM_DOMAIN"
    uaa:
      admin:
        client_secret: PASSWORD
      batch:
        password: PASSWORD
        username: batch_user
      cc:
        client_secret: PASSWORD
      scim:
        userids_enabled: true
        users:
          - name: admin
            password: PASSWORD
            groups:
              - scim.write
              - scim.read
              - openid
              - cloud_controller.admin
              - doppler.firehose
              - routing.router_groups.read
    uaadb:
      address: 10.0.16.101
      databases:
      - {name: uaadb, tag: uaa}
      db_scheme: postgresql
      port: 5524
      roles:
      - {name: uaaadmin, password: PASSWORD, tag: admin}

- name: router_z1
  instances: 1
  resource_pool: small_z1
  templates:
  - {name: gorouter, release: cf}
  - {name: metron_agent, release: cf}
  - {name: consul_agent, release: cf}
  networks:
  - name: cf_private
    static_ips: [10.0.16.102]
  properties:
    dropsonde: {enabled: true}

properties:
  router:
    route_services_secret: PASSWORD
    ssl_skip_validation: true
  networks: {apps: cf_private}
  app_domains: [REPLACE_WITH_APPS_DOMAIN]
  cc:
    allow_app_ssh_access: true
    default_to_diego_backend: true
    internal_api_user: internal_user
    buildpacks:
      blobstore_type: webdav
      webdav_config:
        blobstore_timeout: 5
        password: PASSWORD
        private_endpoint: https://blobstore.service.cf.internal:4443
        public_endpoint: https://blobstore.REPLACE_WITH_SYSTEM_DOMAIN
        secret: PASSWORD
        username: blobstore-username
    droplets:
      blobstore_type: webdav
      webdav_config:
        blobstore_timeout: 5
        password: PASSWORD
        private_endpoint: https://blobstore.service.cf.internal:4443
        public_endpoint: https://blobstore.REPLACE_WITH_SYSTEM_DOMAIN
        secret: PASSWORD
        username: blobstore-username
    external_port: 9022
    packages:
      blobstore_type: webdav
      webdav_config:
        blobstore_timeout: 5
        password: PASSWORD
        private_endpoint: https://blobstore.service.cf.internal:4443
        public_endpoint: https://blobstore.REPLACE_WITH_SYSTEM_DOMAIN
        secret: PASSWORD
        username: blobstore-username
    resource_pool:
      blobstore_type: webdav
      webdav_config:
        blobstore_timeout: 5
        password: PASSWORD
        private_endpoint: https://blobstore.service.cf.internal:4443
        public_endpoint: https://blobstore.REPLACE_WITH_SYSTEM_DOMAIN
        secret: PASSWORD
        username: blobstore-username
    bulk_api_password: PASSWORD
    db_encryption_key: PASSWORD
    default_running_security_groups: [public_networks, dns]
    default_staging_security_groups: [public_networks, dns]
    install_buildpacks:
    - {name: java_buildpack, package: buildpack_java}
    - {name: ruby_buildpack, package: ruby-buildpack}
    - {name: nodejs_buildpack, package: nodejs-buildpack}
    - {name: go_buildpack, package: go-buildpack}
    - {name: python_buildpack, package: python-buildpack}
    - {name: php_buildpack, package: php-buildpack}
    - {name: staticfile_buildpack, package: staticfile-buildpack}
    - {name: binary_buildpack, package: binary-buildpack}
    internal_api_password: PASSWORD
    quota_definitions:
      default:
        memory_limit: 102400
        non_basic_services_allowed: true
        total_routes: 1000
        total_services: -1
    security_group_definitions:
    - name: public_networks
      rules:
      - {destination: 0.0.0.0-9.255.255.255, protocol: all}
      - {destination: 11.0.0.0-169.253.255.255, protocol: all}
      - {destination: 169.255.0.0-172.15.255.255, protocol: all}
      - {destination: 172.32.0.0-192.167.255.255, protocol: all}
      - {destination: 192.169.0.0-255.255.255.255, protocol: all}
    - name: dns
      rules:
      - {destination: 0.0.0.0/0, ports: '53', protocol: tcp}
      - {destination: 0.0.0.0/0, ports: '53', protocol: udp}
    srv_api_uri: https://api.REPLACE_WITH_SYSTEM_DOMAIN
    staging_upload_password: PASSWORD
    staging_upload_user: staging_upload_user
  ccdb:
    address: 10.0.16.101
    databases:
    - {name: ccdb, tag: cc}
    db_scheme: postgres
    port: 5524
    roles:
    - {name: ccadmin, password: PASSWORD, tag: admin}
  consul:
    agent:
      log_level: null
      domain: cf.internal
      servers:
        lan:
        - 10.0.16.105
    encrypt_keys:
    - PASSWORD
    ca_cert: |
      -----BEGIN CERTIFICATE-----
      MIIFBzCCAu+gAwIBAgIBATANBgkqhkiG9w0BAQsFADATMREwDwYDVQQDEwhjb25z
      dWxDQTAeFw0xNjAzMzEyMjAwNDlaFw0yNjAzMzEyMjAwNTJaMBMxETAPBgNVBAMT
      CGNvbnN1bENBMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA0tLZmi6x
      aqbGl7VvKjJoIf2PAilPDzMNH56p7PUOHutRreKn70dzmMr/CbLShMllgjW6ZvOL
      osw0iSVKWewiWtrFD8vmRx+OkmdgxPD9813EvnjsvrGyWOxmK7eTahy2ZpMrzmnV
      RiDNzu0xBlnBHqAFUPOyWLXuBGFGsDgQt7tLlHIp6o0t4qkBjC/qQSKgBcQt9qxq
      cnetiMDDuNhTQT7a+4Br61nwooSAOYmjR9Ox4ULSscMf1qEmQQfJNGJ8ifqF/SOM
      a8Dyt7uUIFRbMzA/wdiy0BXBjJysD5B4tXtWdHSg/8de0STPWOR4dnuOdupAwub+
      9ICOk1FRCEVPHyBVgQppb4SqUa9loJ2fDJQSl0PY+1kC+bC9czEoNg/hDTBYCiVM
      bYPEOnx/810Pm1PoUEmp4rTQUM6EM6pFQBIW3+rxpOJlhxmRJyZzsUpZB9xsh7P2
      Q80OI+XZL/QCI8/t65QamsIbV+IlG5IeALBC+oaC3Rfnfpy3cEXl1vw3Z1qTM9qR
      8ZAUDLUcYcULtDjRqgvEMLaK/vd3UMAXeVpPGgEbno3eXl/spByN2nwmpZM1RcB0
      CtmN+byDv0fAYlUqBsDhes0AaeY3+ZN4opEXUHGR7qDl3hwZBIHG8niFMtVTb5na
      Ktam3U+e1kIVsgbz5+B2KEJKUvjI5yLGDM0CAwEAAaNmMGQwDgYDVR0PAQH/BAQD
      AgEGMBIGA1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFJB06Ynto2S0F7wv5hdE
      y+pVN1NNMB8GA1UdIwQYMBaAFJB06Ynto2S0F7wv5hdEy+pVN1NNMA0GCSqGSIb3
      DQEBCwUAA4ICAQBIMYRZ64GV/oe+yPWK5JhU8W85qT1ROfJC1Hwg6zXtP0pIUlyN
      oAyCtc2i1Z3Cp2K9BIfjg+E2+UhSWnwA/6hLPd0R5KMKNOCYCPZClGlYxskQ6irD
      yiceJGwFj3BJYQzOu+XSn55vvouJOG/I24azkSQ5Fwcvqm+8DdznX5P33PRw1DX7
      mJKNIWflcygrxR908iAIfv0V2btVoDr1YGwPKZKwEJ5jyjrNipbM0kErWzwD6V65
      6qluPalUgUnGrzbAqWgKQ/a015gUqRN1lu6XbRhoV4xCFUFLkubHVoCDJ2Y52pXD
      Tu2WKmwpqx3CI2lTe0Vqs2wrSLZrcKK8t3tUsl3qEPFFdxM8+Q/WIlu6l4USOmAh
      uEmdeSBefFlAYyblOfY84W2h0IsBny4KrBIWl7+QL/38cwA+BII/XoilmoIiDctd
      0aT/55KDrD6kZRR9917Qp586mTagfpHL6SHQbn7hh/YQ0VFlmVffXc9T4mDkcFaR
      04bsl87bgG5Bk4f1msBgmnzvJqzbzdHwVOsjZpeN6N683O6YbxK4DNPqWr/BOdkH
      ZzF06444shzlvEdXlcb122q69AbWNcm+HHkmEQUYH3hoYyiwiicdqDskUScj0QpF
      1ddNutiVjXZsmFRwpFxNBTCoNdns8Q9uUQfzHx8DomWgfX9TNzn7y1cc3g==
      -----END CERTIFICATE-----
    server_cert: |
      -----BEGIN CERTIFICATE-----
      MIIEMDCCAhigAwIBAgIRAOE+FRzzg4N7y23SfTwde6owDQYJKoZIhvcNAQELBQAw
      EzERMA8GA1UEAxMIY29uc3VsQ0EwHhcNMTYwMzMxMjIwMDUyWhcNMTgwMzMxMjIw
      MDUyWjAhMR8wHQYDVQQDExZzZXJ2ZXIuZGMxLmNmLmludGVybmFsMIIBIjANBgkq
      hkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAoiPUUo3sUf8oK98wiAzsEJuhMP1ZW/IG
      PZR3bZyMN4B1WTtSEvATFU0kyCioJv6Num4hBfsYCyUbCX2r4bHwaO1TLQwY+kkT
      phBkhqqsvNk230GNhEC6/oukPlPRjEZau0v0Mzlx1VUHtZCyMZk8pPo6nXxMfapM
      9A7DU2YRjW0LhusGWWEsZVnTgw4A0SYrOA0lutcGkID9fFMOhdrLpk+OPVRPWFae
      Hp4lDLQ6NX78W+u8g6k2ocTjZ2bE4U/87+mYPNgRDJYGkaohOxDNKwDtC97MQmmS
      fdOYapff2NOjGdexWf4Nl2GPm94GDkFQmVpKOuJ5PfMKgFFl4kFBSQIDAQABo3Ew
      bzAOBgNVHQ8BAf8EBAMCA7gwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMC
      MB0GA1UdDgQWBBQJBvWAwnpCTcyTl0uUP9xRTG6IVjAfBgNVHSMEGDAWgBSQdOmJ
      7aNktBe8L+YXRMvqVTdTTTANBgkqhkiG9w0BAQsFAAOCAgEAY1LebPT74T47BbmF
      DY+I3nRgtKWpB97OrtKmKsXjGAd/RlyINmUgQQFlXasTMbUMNz6jHUPMl7o+yrt3
      h8pbBDBmxqQ59cJArHABj77feRptq9tiR21b1lecM7dlf7JtQX4iIMYockEiEZHH
      fA0lzw4r9jyWu8ts7b3Il1TDC0CvCcB6RDfaXLhGZ90BJBx3jh9ekWpT8F25UQwT
      odGq0vSDDdrI/s7NZT3Dw1I7BpLiK2Y9ULR7NDJOZhd885rMZ2Rdt3cb+F7SO+B+
      gE8NktIwkXiMIpdCF0QYSAfi4uZQkdTnL+W7v1lBDPtswuCyozgR3mInVtAeWSPX
      JGDCmiHZFkHhmgoNH15zv0Jx7t9iY5uaDeeNvjbXfFrckMYY6hJf5EhowzTUKOxK
      YpsJuFSFydGxcz/uivEkEl/Gk9FYxrm5FBimV52GI04Y9P3F6O0LfANgjEB1rC2v
      4ebBG13nZddQHbRUzmDemVP42GC68+l2T4eczQB3qhH80HejN7cnzNEgx+24+YwB
      4wtVmSqDh5sB9SlY+HBiJvFgtl0cQISYEnHlxzN/3HbS/xjjXGom9FWXmpTK9FPx
      1SiBHNWnErTnrq7Vxo5rPjzK02xf6CdrUj/35TKau/ssB9IXSLV4mW9z87w4apyI
      WymD9N6OHIM0MEmMcYIqph7Yo2w=
      -----END CERTIFICATE-----
    agent_cert: |
      -----BEGIN CERTIFICATE-----
      MIIEJjCCAg6gAwIBAgIRAJIdGe7fx391grV1/m9i/2AwDQYJKoZIhvcNAQELBQAw
      EzERMA8GA1UEAxMIY29uc3VsQ0EwHhcNMTYwMzMxMjIwMDUzWhcNMTgwMzMxMjIw
      MDUzWjAXMRUwEwYDVQQDEwxjb25zdWwgYWdlbnQwggEiMA0GCSqGSIb3DQEBAQUA
      A4IBDwAwggEKAoIBAQDZ7JoVMCudtjSXp6OGGG2S1xV15R3qm55+Pi4dCgO/g31O
      hJoWQL3fL2S1IfZMEzVjZYRperBhoHHjudkOw8opss2pvHynwQKaj10cKkrI1zak
      t59FkM2GAgkjmeGt3ZbPL+tQM3Pb8K4facxhbM44iOdPpCCGPl/CNWYv8HBtwRUv
      wfTYcmQq5ameA4GEzokKwYXf4cHCMY69sPMKatGldS8B2gal+wBV7PHo+qsvdZug
      YIRLWGbGDBLN9xi2JMHHrNodZKlWSEEEZaSa8kctqFw4iZRYdmKAc7i9HCRxTobP
      W18Dn6GGSCCYMaMCKzrvnwyBWrN5baoNl+cvE7SlAgMBAAGjcTBvMA4GA1UdDwEB
      /wQEAwIDuDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwHQYDVR0OBBYE
      FEjW45GNWfzsd5+P9U1KPlLKKBqeMB8GA1UdIwQYMBaAFJB06Ynto2S0F7wv5hdE
      y+pVN1NNMA0GCSqGSIb3DQEBCwUAA4ICAQDOTO7uujSHt5NUIx8h03TMQxm5Fj1m
      2KaAYmut4HTNBTQHNcvNCTQ86HURj1zaVnNqGNo4zWE89jPb6RkfutcxaWIZzXw1
      CtTFg/DEQlMrRHe7MvaLwr8pVidGjU+yaqDdctzIKi0zKvdb4ImyK3ZucXuoIjth
      S0cMxIBB2D99+8LnPfyAnJT2PFAZ92GyWZRUTZZa3c/QMaJZLaMmp6Yihs/fubQf
      PxgnbirRScO+dqxjzz+3911VHiEGrnLcjE5rvPxjsW4G2g6goK1/Lg+RbAOhAC/6
      UbJQEKAlxqsvBxidkT0OtQJ/AibtPB425KzM2K+v1zXUyFmPHbJspxMCFgkHvOdf
      NYJ6oVLj6lVH5vvFM8pdffbaICZTWBSQG/w5p4EObF5cIL9/PtKGTLj0bi5b8oPc
      ZVRdYiXObpaMO5fJQKY+c3QMzjPczdmBaXF15EyfTUPpRtdyBfica/0BkW0w4XRK
      uEC+YFgx5qEVmscss2K42ZCJ8b/vwcLH0HqMmc8OoPLakVbD/A+64Kt1mdJ7N63K
      GO8DbjsU7qg9T8Y/6O7iJ/FEvvyWmUo0GJlSga/hrHH0LK5ZXVqCjkyVNjojRy0E
      G8X8rON9IPCgIJmpWuOyOKDRONAteTVpYiusmG7s24aX/ZA11vUmpR0iNunbPNct
      bKLaH/vFK5U4cg==
      -----END CERTIFICATE-----
    server_key: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEpQIBAAKCAQEAoiPUUo3sUf8oK98wiAzsEJuhMP1ZW/IGPZR3bZyMN4B1WTtS
      EvATFU0kyCioJv6Num4hBfsYCyUbCX2r4bHwaO1TLQwY+kkTphBkhqqsvNk230GN
      hEC6/oukPlPRjEZau0v0Mzlx1VUHtZCyMZk8pPo6nXxMfapM9A7DU2YRjW0LhusG
      WWEsZVnTgw4A0SYrOA0lutcGkID9fFMOhdrLpk+OPVRPWFaeHp4lDLQ6NX78W+u8
      g6k2ocTjZ2bE4U/87+mYPNgRDJYGkaohOxDNKwDtC97MQmmSfdOYapff2NOjGdex
      Wf4Nl2GPm94GDkFQmVpKOuJ5PfMKgFFl4kFBSQIDAQABAoIBADAJHWY31cOVLHmS
      7fXgni9tbBvvcwHieibUTW2T65al4B5HjNE/fufYqwUBxo+G6sZIyk/TTBRBMfll
      2f5LkUYEyZeW9e9wpvmT8bRT7EkmsTMDYMHFy6CODmLIwlQko8zJe9eRNUBWqKoJ
      7ED1fRoDaEowARlZ0uKbXRLgMmMLam8yWo3aDdeZqp9lQoDLs+R5YubYPSvFkg8l
      19YOrVeXnSeAIKQBx0qLzMWWcYRSFm/WuirhklI1fzrI0RoqpDM7ye7KYjYzxmXM
      a9VukotuMzc316Ny903/QzgriYK04+D2LTTr6MxP5QQLQKpRpnv9BAaOzB7uluo+
      RS5uGAECgYEA111bXtXHmHp69Sw9o+5Jq6LVfg/J0P5VYqy92mAcWfQsyNgSA8nJ
      7q+pEu7e+bRZ/SjqsP9LNm6zF3RIbwDGJUm9dFPT9NCzjAnKDtUcVchL4oodJUKK
      nhKM3DTbr5QEM7AAaaPa/oHc1fw+kMayIhGo/z04xEQuRgF7lUIDPqECgYEAwLuW
      fMq8+GsJB/1eUFwU3k013rnHAxlBC7e+4bdTcUv3QC+bAIGLKkK4bcKn6ElnTrSt
      KXPB7V/Loi0Gwx1035aPwDKVjdQDbug/MX2M/aKeRnQb9VPndluUJDh+deikUAI7
      gqJKv92a279Dat54fjLXKPqsGQAqy403tT6hSakCgYEAkCuO3w19cDWN2lKjcPoz
      lxKKmLk5AQ9BWa0J6wYr9Ivg7xK1/JM4+u/c3y/JVJ/HHhImChbc4rN4cFsHokeC
      XbPff+AeI+USTMzA1u0S6toK8rxCho7k/KyuXzuDVSZhKbjIje+Cyp1kmFskBwb8
      eJIZ78OsHLcHwxV7BZALXAECgYEAnCVevqvifcD6CCcWCjUQEyqqwk/xFGmZcUzk
      sSo9yESrhK0M/1P008BKe2KBdohB0lo/EJ5gN1itOi8Qk3OCBMOOo0BYOhfS0EAJ
      MqdtWvAtGxdmr1PS6uk3FEFQ82YP+WJVpHin5to7ZF2I2UR0ionWF7U/SOIByfgX
      chfTxEECgYEAz8ob55iUxdL/9Dj3JBZp3039l5DqzqS2TVh+b2Gj1R8pvlnHXhBE
      JqXK0KoItB+9qJYzTAfsQJHErPL8+15sqaZHsQrssXgHdGXSqOsTmUWCvNVYi2yr
      0Iy00Tlkozwhwf5Y3PzITI1SWGqa3rZesUydj5FnuyeZTiLVajqPWe4=
      -----END RSA PRIVATE KEY-----
    agent_key: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEowIBAAKCAQEA2eyaFTArnbY0l6ejhhhtktcVdeUd6puefj4uHQoDv4N9ToSa
      FkC93y9ktSH2TBM1Y2WEaXqwYaBx47nZDsPKKbLNqbx8p8ECmo9dHCpKyNc2pLef
      RZDNhgIJI5nhrd2Wzy/rUDNz2/CuH2nMYWzOOIjnT6Qghj5fwjVmL/BwbcEVL8H0
      2HJkKuWpngOBhM6JCsGF3+HBwjGOvbDzCmrRpXUvAdoGpfsAVezx6PqrL3WboGCE
      S1hmxgwSzfcYtiTBx6zaHWSpVkhBBGWkmvJHLahcOImUWHZigHO4vRwkcU6Gz1tf
      A5+hhkggmDGjAis6758MgVqzeW2qDZfnLxO0pQIDAQABAoIBAG3D4PBfLPjpN6BT
      jegTEc3ujB6v4tuyuqg3xZ5W1wB1yH3uCHbA8WIjSwR5MMesvS1tir5eT808tWDQ
      0WXAdGmAaFrgV6FfdGJJZ8qx+q0iyaE54/10LDEdgWDvN18Nx9Jf/pSM9gSIPAwS
      jCFeXpjXTDsvHjq/3BfEMc5fuyFsO6hSncCoY2aXJA6XG6K/qV/ll4djys6GTCDq
      fU2eVWVk2yEM7YfvNocuaocyj+nd8ozDrS2HVtvBMRurjWreNF5zk5F11HaQN/D5
      OfKj8abHZuwlVSqbdaf9mdV+7ENKU1cze82p2ZqwruPGrRFtbezxBBlgEMfzJX3i
      kwa830ECgYEA8wDcoIcVPCAri+Kylwitxevfk2oZcSNqI8q9TAjYDXMfyWISbMt2
      35lq9wVjKx34BSO+BY2HXLnl0sUBdfwWuO4IPzkQtILtwXDdszvAerGgjtZerKXb
      CiLu9ZjpcvlcxPmlitp/WYvX7bJdVsBJQU1ks+5shQLOFv5N4nhG680CgYEA5ZRc
      8gSp4tuSL+1zzBXjdsQfmwkbI6Mw3kZ8e97pBxRtbyH9TyYH3ryAGwOgwFsWuCOW
      Zbqx6CIUr4R20FMBndqe3bPwpL35jdooFMSKaF/a9sDbX3BXYFeJjm2qR9sw8R1E
      461+tuejK18j0DwnzuZHzQ3tT8uTG77ODUQCBDkCgYAxJO15sZgDzuW/pptDnEe4
      jVlr8LswfF8M2gWqiOdY4P1+tszPH97snZRaXMaPg8ITGAVoDhVgFWB7XchL2i2m
      PM2CK8JLH2eCBZdwlhb5OU8lVAlVlT1VMXduR/x+ehve4jYufL3gmD2VHsttrfmi
      sUo6cW+U/to7IDcUJAsDyQKBgQCFQ1u4eJCMyNvQykr/Wm1REYMvIVgJlb7WJ6A2
      3yvxGiBz9AzwFqlW16CdDbwQLE/Bz5aLspV2o+HSCFhXkPdNRAwXsU2ss0Ha35mI
      hJW7BHk75rLwcWum1ulYLbw8PbXpIA5PAvSdA1Sp5m4JgAGzjeR72Ou59/eKkXVW
      KfXpsQKBgG7Lk0TTaeMQ8KtTjp3rYrOCW2S6s6rSUcbTLog7hZELV2/foHftri4p
      bgpHFZuzo7Xt40/6LvM9htBdY9pHtGpBmHnhOqL0dCRB6VSMUZ8wo9ZIANmCxsCI
      S8TIR2w+Bbw6v3yDKwPF6t+86eQvz0/YHbsomgyVPVCcLPkjhBds
      -----END RSA PRIVATE KEY-----

  blobstore:
    admin_users:
    - password: PASSWORD
      username: blobstore-username
    secure_link:
      secret: PASSWORD
    tls:
      cert: |+
        -----BEGIN CERTIFICATE-----
        MIIDQjCCAiqgAwIBAgIJANvIxLqHTfmZMA0GCSqGSIb3DQEBBQUAMEUxCzAJBgNV
        BAYTAkFVMRMwEQYDVQQIEwpTb21lLVN0YXRlMSEwHwYDVQQKExhJbnRlcm5ldCBX
        aWRnaXRzIFB0eSBMdGQwIBcNMTYwMzI1MTgzOTI1WhgPMjI5MDAxMDcxODM5MjVa
        MEwxCzAJBgNVBAYTAlVTMRUwEwYDVQQKEwxDTE9VREZPVU5EUlkxJjAkBgNVBAMT
        HWJsb2JzdG9yZS5zZXJ2aWNlLmNmLmludGVybmFsMIIBIjANBgkqhkiG9w0BAQEF
        AAOCAQ8AMIIBCgKCAQEAxVKn++XNrpKfna39TdysKZvjX3CdBGGfcR0IJVQQQQGi
        iiBQYG2gGeRH6T4Zja90p4EFRzgPnDaUbE8G8dlA7ehI3K83N1xWKiRbUtb5vAgW
        REj2FWDkshGzDX6wV6+N+Ue2yWeEe+N4ojaQRZq8w4rArkUgO3N30eTcrxsMqgBV
        OIru/EMWheuB0SQyq51n2g6Um1F+3pCtusIjJiLTwQ6NF4UQNB0fOqWo6v9fFudi
        g8QSvc8GXVmXvqiQTMSU5/EoWMe06kY8EMJWWZN1Eht43d4QFhRAvhJ7ZicPoIxf
        nWI/3JjBLXJp/HIuu+Rz/KbCDRNSoTux4L1xnLLafQIDAQABoywwKjAoBgNVHREE
        ITAfgh1ibG9ic3RvcmUuc2VydmljZS5jZi5pbnRlcm5hbDANBgkqhkiG9w0BAQUF
        AAOCAQEAO97PryaidEcYcbZTu8E0ikEkbNBzjMY3nfmmdRqAwuxRZAfLwzKbVdrH
        z9eDQLOVAEz9Ftze91IlHpO+KQ3f9khidYXtcEt5j2niOJ81HMo66/lWja27fmpQ
        rfS6YuEleMiTUs8v5URZAgyewb9rlvo06vPW+4GGH6GH/0d30DnFCKp7GMpLEJc+
        z47Jtve2/5xXZKHdTbx2sku9tRRKx2eNSqpB2ev8jwePQ41icDZ4B11AqG2w8Woz
        XL2e5p/TqF9l2rkTcBM8Koi0lz3ZIWpaet6NKvSVQspnLGRIw0I9PvOlKCttNhGT
        cG1R+1W2PzUBtjo477QeJEoYa1k2sQ==
        -----END CERTIFICATE-----
      private_key: |+
        -----BEGIN RSA PRIVATE KEY-----
        MIIEpAIBAAKCAQEAxVKn++XNrpKfna39TdysKZvjX3CdBGGfcR0IJVQQQQGiiiBQ
        YG2gGeRH6T4Zja90p4EFRzgPnDaUbE8G8dlA7ehI3K83N1xWKiRbUtb5vAgWREj2
        FWDkshGzDX6wV6+N+Ue2yWeEe+N4ojaQRZq8w4rArkUgO3N30eTcrxsMqgBVOIru
        /EMWheuB0SQyq51n2g6Um1F+3pCtusIjJiLTwQ6NF4UQNB0fOqWo6v9fFudig8QS
        vc8GXVmXvqiQTMSU5/EoWMe06kY8EMJWWZN1Eht43d4QFhRAvhJ7ZicPoIxfnWI/
        3JjBLXJp/HIuu+Rz/KbCDRNSoTux4L1xnLLafQIDAQABAoIBAAK2eTLAXQyKXYFo
        c/QPFZrY1s5oGPCHew6uDH+e4T5TjG2DtjctKqdQeSCexvEouVzYLD9naOeH5JB8
        oabPitH6gI3wJr0vGswnhc3kwLgyEEROEHwIwfwkvCZyWHBMLJKBxuSL9MlTPkRU
        pbUfRHsXvEBpGOFYXAxZriMGJy1rH4/FfltbPVQX+OadL/bJE4DC/5GXnLV3ft3K
        BVVZyvafhhLCrm24mNgSxHziuAG10UXk+QmjJmzj0CgqAIeLoI7+hCz7nf0puzkO
        UAsxYMh5GJqSEmWtnVyk1PbhA7XBK1zRCciMcy9U+IZ3yydH+HT40+ht8T8Wyop2
        Hj9jkCECgYEA7vsrzK/FV+NGhceANZdRhD35cKTr5hzLOO0J+NJVvKjHEyelgOWP
        +5WL+ahuhKCvlPuCXl/lfEq7PQrpSrqs3psgDi22oMGX4P/FXjlpmwFMJM49RcYk
        azBJLGj1/1F0VkG4WMSieFQ+uCZc6Q/rm/EdOyCsn4acw8SCmKuIvEkCgYEA02AF
        B1lkQzqRYZbT15T+zuoTV4GRjRXQugzT+aLAyI6fW70VfZOWWtwMK86VJVmSKRqD
        W4uaiKQGVQaq5szc3i+Hv6xndj7sL6e1VypFtAjjeswY/GPSjrt53SKz4TWOfCZq
        8JZVQmAebPPTdjyO8DZcGb6y8C7IhKByVdCLJJUCgYAYQa5EbGLfdNYnpgRBbEZ9
        4bx7zoGTLcEC2ix08QR6zbbHHvMRjjt7EcbPZGUzWQv5Vz34TkuAviUbIQxk5WW+
        gohSaBltX7kGwW9LDRDHBu6vna9icaYoqxICS/UMITxptOn9OJg1Fnf3QQ2VKmSD
        w4lwAvUCjCtFQ6Dt1hte4QKBgQCQCnriyzPb7Glty06JNmt9rV2I4C7Dqf4XCu7Y
        yuP8x9Qou+2NKanoONPCdoCEd0l24S5qj/O68auu/WAw76IDdvhW0bGfjrl8sBiP
        Uas2SGhcIgFU3OF7ip48540VB14VlEiDsq5fEQkqze1oQVRWtXSFxsJBkl/qoTvI
        5tgrEQKBgQDCKTdsTsTK+UAhLB45rVg6IppDoY/X8qspaNtYYHa4bG4X9qk1Gl+2
        fPuTr6OlWPa4VA4wb1DuDIk3gpdBIJWo3Jv7miUeo6CgRUalZ6kN6PpBtiY3lKK0
        7sTV9KfdlETLUwh8qkEsoMJCpNzvRvRVQn5Xp25W9ssnVnXgY7qpQg==
        -----END RSA PRIVATE KEY-----
      ca_cert: |
        -----BEGIN CERTIFICATE-----
        MIIDtzCCAp+gAwIBAgIJAIPgaUgWRCE8MA0GCSqGSIb3DQEBBQUAMEUxCzAJBgNV
        BAYTAkFVMRMwEQYDVQQIEwpTb21lLVN0YXRlMSEwHwYDVQQKExhJbnRlcm5ldCBX
        aWRnaXRzIFB0eSBMdGQwIBcNMTYwMzI1MTgzOTI1WhgPMjI5MDAxMDcxODM5MjVa
        MEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIEwpTb21lLVN0YXRlMSEwHwYDVQQKExhJ
        bnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
        ggEKAoIBAQDDmSpF1agfBnLiyi+KlbqvetxxFqG0ONDOZQTiTnc/xkmwJdCsEwWe
        CA8okKm6aB9/C5IQz8//7KnZeAWghl+PCVanr+Ax1vjZKLPO2Ccin7oQ7wVSqJ5q
        5slVK2nfRtLcyXdb5rPIlNEMRpupv0jiRAaOoh5KvOSWA770zIJF3qP9IyuA8P6y
        XCU4+lu6XOeaHWRCXyzpjBGgnc6M/kYx8sKp6ktu1dXYUpNld4ICjSbmLDdq6hd6
        OU7WkR3/AZQ3q3F7hiRg7CVAea24RsV30a2mYgVZnUgA8A12zuHXV0S+955CUntr
        Cygv4JovFCf/VaBuUG7LGJNuCy+rJrBRAgMBAAGjgacwgaQwHQYDVR0OBBYEFDcq
        5thPoSs0QiQLQMmW2WxjEVSSMHUGA1UdIwRuMGyAFDcq5thPoSs0QiQLQMmW2Wxj
        EVSSoUmkRzBFMQswCQYDVQQGEwJBVTETMBEGA1UECBMKU29tZS1TdGF0ZTEhMB8G
        A1UEChMYSW50ZXJuZXQgV2lkZ2l0cyBQdHkgTHRkggkAg+BpSBZEITwwDAYDVR0T
        BAUwAwEB/zANBgkqhkiG9w0BAQUFAAOCAQEANqwgDsDrQc3iKMsC8/HoCOKz/Gu7
        EHDncfeuo8/+6vuHf5IsJrkpzNoFI7OWMp/PsDQ2L7jAFATKYNhpbwJwAknSlTBk
        2Pi2j303EGdLLGFdhKWK46MUtBMf6KBfUn/ReFgTUIH+NrX/mnr3ZdK/O2DfIjYj
        Wdc+6NNVewmA0E/n2O4BasfsIH9xZ8Sjt1yeyyW/x47i1UIgsSSg7J6h3GEfG1PA
        a2avU7gd1ERzy4z0FobouMTCiQ+BDfQHT2Jg8WLbdyEj/TH56w4cF29pQ0coZOC1
        bAzM00WOuWYAt8wwcB2+feZiOUxoMzE13XdDf+XaJwmBmKEKYf56+kJTkQ==
        -----END CERTIFICATE-----
  databases:
    databases:
    - {name: ccdb, tag: cc, citext: true}
    - {name: uaadb, tag: uaa, citext: true}
    port: 5524
    roles:
    - {name: ccadmin, password: PASSWORD, tag: admin}
    - {name: uaaadmin, password: PASSWORD, tag: admin}
  description: Cloud Foundry sponsored by Pivotal
  domain: REPLACE_WITH_SYSTEM_DOMAIN
  etcd:
    advertise_urls_dns_suffix: etcd.service.cf.internal
    cluster:
    - name: diego_z1
      instances: 1
    machines: ["etcd.service.cf.internal"]
    peer_require_ssl: false
    require_ssl: false

  doppler:
    port: 4443
  logger_endpoint:
    port: 4443
  loggregator:
    etcd:
      machines: [10.0.16.104]
  loggregator_endpoint:
    shared_secret: PASSWORD
  metron_agent:
    zone: z1
    deployment: minimal-aws
    dropsonde_incoming_port: 3457
  metron_endpoint:
    shared_secret: PASSWORD
  nats:
    machines: [10.0.16.103]
    password: PASSWORD
    port: 4222
    user: nats
  ssl:
    skip_cert_verify: true
  system_domain: REPLACE_WITH_SYSTEM_DOMAIN
  system_domain_organization: default_organization
  uaa:
    clients:
      cf:
        access-token-validity: 600
        authorities: uaa.none
        authorized-grant-types: implicit,password,refresh_token
        autoapprove: true
        override: true
        refresh-token-validity: 2592000
        scope: cloud_controller.read,cloud_controller.write,openid,password.write,cloud_controller.admin,scim.read,scim.write,doppler.firehose,uaa.user,routing.router_groups.read
      cc-service-dashboards:
        authorities: clients.read,clients.write,clients.admin
        authorized-grant-types: client_credentials
        scope: openid,cloud_controller_service_permissions.read
        secret: PASSWORD
      cloud_controller_username_lookup:
        authorities: scim.userids
        authorized-grant-types: client_credentials
        secret: PASSWORD
      cc_routing:
        authorities: routing.router_groups.read
        secret: PASSWORD
        authorized-grant-types: client_credentials
      gorouter:
        authorities: routing.routes.read
        authorized-grant-types: client_credentials,refresh_token
        secret: PASSWORD
      tcp_emitter:
        authorities: routing.routes.write,routing.routes.read
        authorized-grant-types: client_credentials,refresh_token
        secret: PASSWORD
      tcp_router:
        authorities: routing.routes.read
        authorized-grant-types: client_credentials,refresh_token
        secret: PASSWORD
      doppler:
        authorities: uaa.resource
        secret: PASSWORD
      login:
        authorities: oauth.login,scim.write,clients.read,notifications.write,critical_notifications.write,emails.write,scim.userids,password.write
        authorized-grant-types: authorization_code,client_credentials,refresh_token
        redirect-uri: https://login.REPLACE_WITH_SYSTEM_DOMAIN
        scope: openid,oauth.approvals
        secret: PASSWORD
      servicesmgmt:
        authorities: uaa.resource,oauth.service,clients.read,clients.write,clients.secret
        authorized-grant-types: authorization_code,client_credentials,password,implicit
        autoapprove: true
        redirect-uri: https://servicesmgmt.REPLACE_WITH_SYSTEM_DOMAIN/auth/cloudfoundry/callback
        scope: openid,cloud_controller.read,cloud_controller.write
        secret: PASSWORD

    jwt:
      signing_key: |
        -----BEGIN RSA PRIVATE KEY-----
        MIICXAIBAAKBgQDHFr+KICms+tuT1OXJwhCUmR2dKVy7psa8xzElSyzqx7oJyfJ1
        JZyOzToj9T5SfTIq396agbHJWVfYphNahvZ/7uMXqHxf+ZH9BL1gk9Y6kCnbM5R6
        0gfwjyW1/dQPjOzn9N394zd2FJoFHwdq9Qs0wBugspULZVNRxq7veq/fzwIDAQAB
        AoGBAJ8dRTQFhIllbHx4GLbpTQsWXJ6w4hZvskJKCLM/o8R4n+0W45pQ1xEiYKdA
        Z/DRcnjltylRImBD8XuLL8iYOQSZXNMb1h3g5/UGbUXLmCgQLOUUlnYt34QOQm+0
        KvUqfMSFBbKMsYBAoQmNdTHBaz3dZa8ON9hh/f5TT8u0OWNRAkEA5opzsIXv+52J
        duc1VGyX3SwlxiE2dStW8wZqGiuLH142n6MKnkLU4ctNLiclw6BZePXFZYIK+AkE
        xQ+k16je5QJBAN0TIKMPWIbbHVr5rkdUqOyezlFFWYOwnMmw/BKa1d3zp54VP/P8
        +5aQ2d4sMoKEOfdWH7UqMe3FszfYFvSu5KMCQFMYeFaaEEP7Jn8rGzfQ5HQd44ek
        lQJqmq6CE2BXbY/i34FuvPcKU70HEEygY6Y9d8J3o6zQ0K9SYNu+pcXt4lkCQA3h
        jJQQe5uEGJTExqed7jllQ0khFJzLMx0K6tj0NeeIzAaGCQz13oo2sCdeGRHO4aDh
        HH6Qlq/6UOV5wP8+GAcCQFgRCcB+hrje8hfEEefHcFpyKH+5g1Eu1k0mLrxK2zd+
        4SlotYRHgPCEubokb2S1zfZDWIXW3HmggnGgM949TlY=
        -----END RSA PRIVATE KEY-----

      verification_key: |
        -----BEGIN PUBLIC KEY-----
        MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDHFr+KICms+tuT1OXJwhCUmR2d
        KVy7psa8xzElSyzqx7oJyfJ1JZyOzToj9T5SfTIq396agbHJWVfYphNahvZ/7uMX
        qHxf+ZH9BL1gk9Y6kCnbM5R60gfwjyW1/dQPjOzn9N394zd2FJoFHwdq9Qs0wBug
        spULZVNRxq7veq/fzwIDAQAB
        -----END PUBLIC KEY-----

    ssl:
      port: -1
    url: https://uaa.REPLACE_WITH_SYSTEM_DOMAIN
  capi:
    nsync:
      bbs:
        ca_cert: ""
        client_cert: ""
        client_key: ""
        require_ssl: false
      cc:
        base_url: https://api.REPLACE_WITH_SYSTEM_DOMAIN
        basic_auth_password: PASSWORD
      diego_privileged_containers: true
    tps:
      bbs:
        ca_cert: ""
        client_cert: ""
        client_key: ""
        require_ssl: false
      cc:
        base_url: https://api.REPLACE_WITH_SYSTEM_DOMAIN
        basic_auth_password: PASSWORD
      traffic_controller_url: wss://doppler.REPLACE_WITH_SYSTEM_DOMAIN:4443
    tps_listener:
      bbs:
        ca_cert: ""
        client_cert: ""
        client_key: ""
        require_ssl: false
      cc:
        base_url: https://api.REPLACE_WITH_SYSTEM_DOMAIN
        basic_auth_password: PASSWORD
    stager:
      bbs:
        ca_cert: ""
        client_cert: ""
        client_key: ""
        require_ssl: false
      cc:
        base_url: https://api.REPLACE_WITH_SYSTEM_DOMAIN
        basic_auth_password: PASSWORD
  diego:
    auctioneer:
      bbs:
        ca_cert: ""
        client_cert: ""
        client_key: ""
        require_ssl: false
    bbs:
      active_key_label: active
      encryption_keys:
      - label: active
        passphrase: PASSWORD
      ca_cert: ""
      etcd:
        ca_cert: ""
        client_cert: ""
        client_key: ""
        require_ssl: false
      require_ssl: false
      server_cert: ""
      server_key: ""
    converger:
      bbs:
        ca_cert: ""
        client_cert: ""
        client_key: ""
        require_ssl: false
    rep:
      bbs:
        ca_cert: ""
        client_cert: ""
        client_key: ""
        require_ssl: false
      preloaded_rootfses: ["cflinuxfs2:/var/vcap/packages/cflinuxfs2/rootfs"]
    executor:
      memory_capacity_mb: 30720
      disk_capacity_mb: 163840
    route_emitter:
      bbs:
        ca_cert: ""
        client_cert: ""
        client_key: ""
        require_ssl: false
      nats:
        machines: [10.0.16.103]
        password: PASSWORD
        port: 4222
        user: nats
    ssl:
      skip_cert_verify: true
  garden:
    graph_cleanup_threshold_in_mb: 0
    persistent_image_list: ["/var/vcap/packages/cflinuxfs2/rootfs"]
    deny_networks:
    - 0.0.0.0/0

# code_snippet cf-minimal-aws end
# The previous line helps maintain current documentation at http://docs.cloudfoundry.org.

REPLACE_WITH_PRIVATE_SUBNET_ID ... cf_privateのサブネットID
REPLACE_WITH_PUBLIC_SUBNET_ID ... cf_publicのサブネットID
REPLACE_WITH_ELASTIC_IP ... HA_PROXYのEIP
REPLACE_WITH_SSL_CERT_AND_KEY ... 次に説明
REPLACE_WITH_SYSTEM_DOMAIN ... 上記の例ではsystem.awscf.ik.am。自分のドメインに合わせてください
REPLACE_WITH_APPS_DOMAIN ... 上記の例ではapps.awscf.ik.am。自分のドメインに合わせてください
PASSWORD ... 変更した方が良いです

REPLACE_WITH_SSL_CERT_AND_KEYは次の出力結果をコピー&ペーストしてください。インデントに注意。

openssl genrsa -out cf.key 1024
echo -e "JP\nTokyo\nCity\nCompany\nSection\n*.system.awscf.ik.am\nmaking@example.com\n\n" | openssl req -new -key cf.key -out cf.csr
openssl x509 -req -in cf.csr -signkey cf.key -out cf.crt
cat cf.crt cf.key

ここまで設定できたらようやくマニフェストファイルが完成です。

BOSHのStemcell(VMのベーステンプレート)とリリース(ソフトウェア+スクリプト群)をBOSH Directorにアップロードして、bosh deploy！

wget https://bosh.io/d/stemcells/bosh-aws-xen-hvm-ubuntu-trusty-go_agent?v=3262.14 -O bosh-aws-xen-hvm-ubuntu-trusty-go_agent-3262.14.tgz
wget https://bosh.io/d/github.com/cloudfoundry/cf-release?v=243 -O cf-release-243.tgz
wget https://bosh.io/d/github.com/cloudfoundry/diego-release?v=0.1485.0 -O diego-release-0.1485.0.tgz
wget https://bosh.io/d/github.com/cloudfoundry/garden-linux-release?v=0.342.0 -O garden-linux-release-0.342.0.tgz
wget https://bosh.io/d/github.com/cloudfoundry-incubator/etcd-release?v=67 -O etcd-release-67.tgz
wget https://bosh.io/d/github.com/cloudfoundry/cflinuxfs2-rootfs-release?v=1.29.0 -O cflinuxfs2-rootfs-release-1.29.0.tgz

bosh upload stemcell bosh-aws-xen-hvm-ubuntu-trusty-go_agent-3262.14.tgz
bosh upload release cf-release-243.tgz
bosh upload release diego-release-0.1485.0.tgz
bosh upload release garden-linux-release-0.342.0.tgz
bosh upload release etcd-release-67.tgz
bosh upload release cflinuxfs2-rootfs-release-1.29.0.tgz

bosh deployment aws-cf.yml
bosh -n deploy

30〜40分くらいで完了すると思います。

EC2一覧を見ると次のようになっていると思います。

EIP一覧を見るとHA_PROXYのインスタンスにアサインされていますね。

疎通確認

$ curl http://<System Domain>/v2/info
{"name":"","build":"","support":"","version":0,"description":"Cloud Foundry sponsored by Pivotal","authorization_endpoint":"https://login.<System Domain>","token_endpoint":"https://uaa.<System Domain>","min_cli_version":null,"min_recommended_cli_version":null,"api_version":"2.62.0","app_ssh_endpoint":"ssh.<System Domain>:2222","app_ssh_host_key_fingerprint":null,"app_ssh_oauth_client":"ssh-proxy","logging_endpoint":"wss://loggregator.<System Domain>:4443","doppler_logging_endpoint":"wss://doppler.<System Domain>:4443"}

OKであれば、

CF CLIでcf loginです。

$ cf login -a api.<System Domain> --skip-ssl-validation -u admin -p <PASSWORD>
$ cf create-space demo
$ cf target -s demo
$ cd /tmp
$ git clone https://github.com/making/hacker-tackle-demo.git
$ cd hacker-tackle-demo/hacker-tackle
$ cf push
$ curl -i hacker-tackle.<Apps Domain>
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Date: Fri, 23 Sep 2016 10:14:41 GMT
X-Vcap-Request-Id: 60cae1e3-cd1c-4bab-6f6c-157ce234af80
Content-Length: 1234
Connection: keep-alive

0️⃣   0️⃣   0️⃣ 0️⃣     0️⃣ 0️⃣ 0️⃣  0️⃣    0️⃣  0️⃣ 0️⃣ 0️⃣   0️⃣ 0️⃣ 0️⃣   
0️⃣   0️⃣  0️⃣   0️⃣   0️⃣       0️⃣   0️⃣   0️⃣       0️⃣    0️⃣  
0️⃣ 0️⃣ 0️⃣  0️⃣ 0️⃣ 0️⃣   0️⃣       0️⃣ 0️⃣     0️⃣ 0️⃣     0️⃣ 0️⃣ 0️⃣  
0️⃣   0️⃣  0️⃣    0️⃣  0️⃣       0️⃣   0️⃣   0️⃣       0️⃣    0️⃣  
0️⃣   0️⃣  0️⃣    0️⃣   0️⃣ 0️⃣ 0️⃣  0️⃣    0️⃣  0️⃣ 0️⃣ 0️⃣   0️⃣    0️⃣  
                                                 
0️⃣ 0️⃣ 0️⃣ 0️⃣   0️⃣ 0️⃣     0️⃣ 0️⃣ 0️⃣  0️⃣    0️⃣  0️⃣       0️⃣ 0️⃣ 0️⃣  
   0️⃣     0️⃣   0️⃣   0️⃣       0️⃣   0️⃣   0️⃣       0️⃣ 
   0️⃣     0️⃣ 0️⃣ 0️⃣   0️⃣       0️⃣ 0️⃣     0️⃣       0️⃣ 0️⃣    
   0️⃣     0️⃣    0️⃣  0️⃣       0️⃣   0️⃣   0️⃣       0️⃣  
   0️⃣     0️⃣    0️⃣   0️⃣ 0️⃣ 0️⃣  0️⃣    0️⃣  0️⃣ 0️⃣ 0️⃣   0️⃣ 0️⃣ 0️⃣

できた。

HA Proxyの代わりにELBを使用

HA Proxyは通常使用しないので、ELBに変更します。

基本的には

https://docs.pivotal.io/pivotalcf/1-7/customizing/pcf-aws-manual-config.html#pcfaws-lb

参照。cf-elbという名前でELB作成。サブネットはcf-publicを選択。

セキュリティグループはそのまま。

HA Proxy設定時に作ったcf.keyとcf.crtの内容を貼り付けて新規Certification作成。

ヘルスチェックはTCP:80で。

振り先のインスタンスは設定しなくてOK(BOSHが設定する)。

確認して"Create"をクリック。

AWS側の問題か？しばしばCertificateの作成に失敗します。

自分は戻ってやり直したらうまく行きました。

うまくいかなかったらAWS CLIでServer Certificateを作成してexisting certificateとして選択すると確実です。

作成されたELBのDNS nameをコピーして、

*.awscfのCNAMEレコードに登録します。先ほどHA Proxy用に設定したものは削除してください。

aws-cf.ymlのうち次の箇所を変更してください。

resource_pools:
- name: small_z1
  network: cf_private
  stemcell:
    name: bosh-aws-xen-hvm-ubuntu-trusty-go_agent
    version: latest
  cloud_properties:
    availability_zone: ap-northeast-1a
    instance_type: t2.medium
## ここから追加
- name: router_z1
  network: cf_private
  stemcell:
    name: bosh-aws-xen-hvm-ubuntu-trusty-go_agent
    version: latest
  cloud_properties:
    availability_zone: ap-northeast-1a
    instance_type: t2.medium
    elbs:
    - cf-elb
## ここまで
## ...

と

- name: ha_proxy_z1
  instances: 0 # <- 1から変更
  resource_pool: small_z1
  templates:
  - {name: haproxy, release: cf}
  - {name: consul_agent, release: cf}
  - {name: metron_agent, release: cf}
  networks:
  # 次の2行をコメントアウト
  # - name: elastic
  #   static_ips: [YOUR_ELASTIC_IP]
## ...

と

- name: router_z1
  instances: 1
  resource_pool: router_z1 # <- small_z1から変更
## ...

変更したらbosh deploy！

$ bosh deploy
Acting as user 'admin' on deployment 'cf' on 'my-bosh'
Getting deployment properties from director...

Detecting deployment changes
----------------------------
resource_pools:
- name: router_z1
  network: cf_private
  stemcell:
    name: bosh-aws-xen-hvm-ubuntu-trusty-go_agent
    version: '3262.14'
  cloud_properties:
    availability_zone: ap-northeast-1a
    instance_type: t2.medium
    elbs:
    - cf-elb

jobs:
- name: ha_proxy_z1
  instances: 1
  instances: 0
  networks:
  - name: elastic
    static_ips:
    - 52.193.13.247
- name: router_z1
  resource_pool: small_z1
  resource_pool: router_z1
Please review all changes carefully

Deploying
---------
Are you sure you want to deploy? (type 'yes' to continue):  yes

  Started preparing deployment > Preparing deployment. Done (00:00:01)

  Started preparing package compilation > Finding packages to compile. Done (00:00:00)

  Started deleting unneeded instances ha_proxy_z1 > ha_proxy_z1/0 (6e376335-d587-41ef-8768-687eddbfb6cc)

  Started updating job router_z1 > router_z1/0 (3a3ab5e1-e8ad-49eb-b4fe-a9cec6841186) (canary)

     Done deleting unneeded instances ha_proxy_z1 > ha_proxy_z1/0 (6e376335-d587-41ef-8768-687eddbfb6cc) (00:01:10)

     Done updating job router_z1 > router_z1/0 (3a3ab5e1-e8ad-49eb-b4fe-a9cec6841186) (canary) (00:03:20)

Task 17 done

Started		2016-09-23 03:13:49 UTC
Finished	2016-09-23 03:17:12 UTC
Duration	00:03:23

Deployed 'cf' to 'my-bosh'

成功したらHA ProxyのVMが削除されたことが確認できます。

ELBのインスタンス一覧にRouterが登録されていることも確認できます。

あとは先ほどと同様に動作確認して問題なければOK。

TODO

cf ssh(Diego Brain)のためのELB設定
blobstoreをWebDAVじゃなくてS3に変更
Multi-AZ対応 (きっと書かない)

Platformをアップロードする方法も書かないとねぇ。あと、Cell、Router、Loggregatorのinstance数は増やしましょう。

IK.AM