前記事でVirtualBoxにBOSH v2 CLIを使ってBOSH Liteをインストールした。
以下の記事はこの設定が済んでいる前提である。
今度はこのBOSH Lite上にConcourseをインストールする。
最終的に作りたいのは次のような環境。
まずはHA Proxy抜きな状態からセットアップする。
目次
Cloud Configの修正
ConcourseのWEB UIをStatic IPで固定するためにcloud configを修正する。
次のコマンドでcloud-config.ymlをダウンロード。
bosh2 cloud-config > ~/cloud-config.yml
Static IPに10.244.0.200を追下。
networks:
- name: default
subnets:
- azs:
- z1
- z2
- z3
gateway: 10.244.0.1
range: 10.244.0.0/24
reserved: []
static:
- 10.244.0.34
- 10.244.0.200 # ここ追加
type: manual
適用。
bosh2 update-cloud-config ~/cloud-config.yml
Concourseのデプロイ
manifestはこちらに穴埋め状態で用意しておいた。
Concourseに必要なBOSH Releaseのurlはmanifestに記載してあり、明示的にbosh upload-releaseをする必要はない。
wget https://raw.githubusercontent.com/making/bosh-manifests/master/concourse.yml
BOSH v2 CLIの機能でpasswordやTLS証明書といったcredentials情報は自動生成させ、manifest上には記述しない。
生成されたcredentials情報は次のbosh2 interpolateコマンドで確認できる。
bosh2 interpolate concourse.yml --vars-store ./creds-concourse.yml -v internal_ip=10.244.0.200
creds-concourse.ymlに生成されたcredentials情報が保存される。
bosh2 deploy -d concourse --vars-store ./creds-concourse.yml -v internal_ip=10.244.0.200 concourse.yml
ログは次の通り。
$ bosh2 deploy -d concourse --vars-store ./creds-concourse.yml -v internal_ip=10.244.0.200 concourse.yml
Using environment '192.168.50.6' as client 'admin'
Using deployment 'concourse'
Release 'concourse/2.7.3' already exists.
Release 'garden-runc/1.4.0' already exists.
+ azs:
+ - name: z1
+ - name: z2
+ - name: z3
+ vm_types:
+ - name: default
+ compilation:
+ az: z1
+ network: default
+ reuse_compilation_vms: true
+ vm_type: default
+ workers: 5
+ networks:
+ - name: default
+ subnets:
+ - azs:
+ - z1
+ - z2
+ - z3
+ gateway: 10.244.0.1
+ range: 10.244.0.0/24
+ reserved: []
+ static:
+ - 10.244.0.34
+ - 10.244.0.200
+ type: manual
+ disk_types:
+ - disk_size: 1024
+ name: default
+ stemcells:
+ - alias: trusty
+ os: ubuntu-trusty
+ version: '3363.19'
+ releases:
+ - name: concourse
+ sha1: ca38944353c893660a428cc6678941734b51b036
+ url: https://bosh.io/d/github.com/concourse/concourse?v=2.7.3
+ version: 2.7.3
+ - name: garden-runc
+ sha1: 1d6020e761806d7f355ceda06c889c582b47dc32
+ url: https://bosh.io/d/github.com/cloudfoundry/garden-runc-release?v=1.4.0
+ version: 1.4.0
+ update:
+ canaries: 1
+ canary_watch_time: 1000-60000
+ max_in_flight: 1
+ serial: false
+ update_watch_time: 1000-60000
+ instance_groups:
+ - azs:
+ - z1
+ instances: 1
+ jobs:
+ - name: atc
+ properties:
+ basic_auth_password: "<redacted>"
+ basic_auth_username: "<redacted>"
+ bind_port: "<redacted>"
+ external_url: "<redacted>"
+ postgresql_database: "<redacted>"
+ tls_bind_port: "<redacted>"
+ tls_cert: "<redacted>"
+ tls_key: "<redacted>"
+ release: concourse
+ - name: tsa
+ properties: {}
+ release: concourse
+ name: web
+ networks:
+ - name: default
+ static_ips:
+ - 10.244.0.200
+ stemcell: trusty
+ vm_type: default
+ - azs:
+ - z1
+ instances: 1
+ jobs:
+ - name: postgresql
+ properties:
+ databases:
+ - name: "<redacted>"
+ password: "<redacted>"
+ role: "<redacted>"
+ release: concourse
+ name: db
+ networks:
+ - name: default
+ persistent_disk_type: default
+ stemcell: trusty
+ vm_type: default
+ - azs:
+ - z1
+ instances: 1
+ jobs:
+ - name: groundcrew
+ properties: {}
+ release: concourse
+ - name: baggageclaim
+ properties: {}
+ release: concourse
+ - name: garden
+ properties:
+ garden:
+ listen_address: "<redacted>"
+ listen_network: "<redacted>"
+ release: garden-runc
+ name: worker
+ networks:
+ - name: default
+ stemcell: trusty
+ vm_type: default
+ name: concourse
+ variables:
+ - name: default_ca
+ options:
+ common_name: ca
+ is_ca: true
+ type: certificate
+ - name: atc_ssl
+ options:
+ alternative_names:
+ - 10.244.0.200
+ ca: default_ca
+ common_name: 10.244.0.200
+ type: certificate
+ - name: postgres_password
+ type: password
+ - name: ui_password
+ type: password
Continue? [yN]: y
Task 11
20:29:07 | Preparing deployment: Preparing deployment (00:00:00)
20:29:07 | Preparing package compilation: Finding packages to compile (00:00:00)
20:29:07 | Compiling packages: tar/f2ea61c537d8eb8cb2d691ce51e8516b28fa5bb7
20:29:07 | Compiling packages: busybox/fc652425c32d0dad62f45bca18e1899671e2e570
20:29:07 | Compiling packages: shadow/7a5e46357a33cafc8400a8e3e2e1f6d3a1159cb6
20:29:07 | Compiling packages: libseccomp/7a54b27a61b42980935e863d7060dc5a076b44d0
20:29:07 | Compiling packages: apparmor/d2a9bb24b85a144e874e7627a5fefb4e9b8b30f3
20:29:17 | Compiling packages: busybox/fc652425c32d0dad62f45bca18e1899671e2e570 (00:00:10)
20:29:17 | Compiling packages: iptables/70cd40ad87371de73d6bdfc34967e94422fc2cc4
20:29:29 | Compiling packages: libseccomp/7a54b27a61b42980935e863d7060dc5a076b44d0 (00:00:22)
20:29:29 | Compiling packages: golang/63a243be32451af083a062ba2c929c3f2b34f132
20:29:46 | Compiling packages: shadow/7a5e46357a33cafc8400a8e3e2e1f6d3a1159cb6 (00:00:39)
20:29:46 | Compiling packages: btrfs_tools/6856973d0bc2dc673b6740f5e164ba77a77fd4a6 (00:00:01)
20:29:47 | Compiling packages: vagrant_cloud_resource/965f230db2311645cbbf9d240db6ee5fdb43a983 (00:00:02)
20:29:49 | Compiling packages: tracker_resource/caedeb694f9b4945b12a3ce9731d5cfbf98a0e56
20:29:51 | Compiling packages: golang/63a243be32451af083a062ba2c929c3f2b34f132 (00:00:22)
20:29:51 | Compiling packages: time_resource/9c9481e92eeb038d3621854e325412fdea5a6511 (00:00:02)
20:29:53 | Compiling packages: semver_resource/d012d4c6d21de36c3a576b20dfc3b134d10ec42c
20:29:53 | Compiling packages: tracker_resource/caedeb694f9b4945b12a3ce9731d5cfbf98a0e56 (00:00:04)
20:29:53 | Compiling packages: s3_resource/2af4d1cea8d5b617e6d4f580818b414db7e54d42 (00:00:05)
20:29:58 | Compiling packages: pool_resource/7bb393617fa62ee257c4e56c1660cbf99fa7ee62
20:29:58 | Compiling packages: semver_resource/d012d4c6d21de36c3a576b20dfc3b134d10ec42c (00:00:05)
20:29:58 | Compiling packages: hg_resource/ad85955734d8ce0a813218102dcc4d339d779ecc
20:29:59 | Compiling packages: apparmor/d2a9bb24b85a144e874e7627a5fefb4e9b8b30f3 (00:00:52)
20:29:59 | Compiling packages: github_release_resource/905a156da0707f87155b1390c05722d3e6a7532f
20:30:01 | Compiling packages: iptables/70cd40ad87371de73d6bdfc34967e94422fc2cc4 (00:00:44)
20:30:01 | Compiling packages: git_resource/9eab8c35f89630fefce5e3bcd9cb39697ee835c0
20:30:02 | Compiling packages: pool_resource/7bb393617fa62ee257c4e56c1660cbf99fa7ee62 (00:00:04)
20:30:02 | Compiling packages: docker_image_resource/b7ad85a4aa0e63bb89fc58682de209429a0db1ef
20:30:02 | Compiling packages: github_release_resource/905a156da0707f87155b1390c05722d3e6a7532f (00:00:03)
20:30:02 | Compiling packages: cf_resource/6c95b8277b3310d3f4fee49efdca7b7b61dbd602
20:30:03 | Compiling packages: hg_resource/ad85955734d8ce0a813218102dcc4d339d779ecc (00:00:05)
20:30:03 | Compiling packages: bosh_io_stemcell_resource/4aa948b8442e7f2395984931398d330000c4604f
20:30:05 | Compiling packages: git_resource/9eab8c35f89630fefce5e3bcd9cb39697ee835c0 (00:00:04)
20:30:05 | Compiling packages: bosh_io_release_resource/980773e5718d0f4e80778735af1560c0438ccfe6
20:30:06 | Compiling packages: bosh_io_stemcell_resource/4aa948b8442e7f2395984931398d330000c4604f (00:00:03)
20:30:06 | Compiling packages: bosh_deployment_resource/56c0e245445bab5097dd6725fd99e98fa638b130
20:30:07 | Compiling packages: bosh_io_release_resource/980773e5718d0f4e80778735af1560c0438ccfe6 (00:00:02)
20:30:07 | Compiling packages: archive_resource/d95bdd7a1d7ab0467da4645786e0e2962e3687c7
20:30:07 | Compiling packages: cf_resource/6c95b8277b3310d3f4fee49efdca7b7b61dbd602 (00:00:05)
20:30:07 | Compiling packages: postgresql_9.3/f8b7a26f105b921843fa5565818b50cc45ace6af
20:30:09 | Compiling packages: archive_resource/d95bdd7a1d7ab0467da4645786e0e2962e3687c7 (00:00:02)
20:30:09 | Compiling packages: generated_worker_key/00b5b02050bc6588cdfe9e523b2ba24a6c9de3c7 (00:00:01)
20:30:10 | Compiling packages: generated_tsa_host_key/00b5b02050bc6588cdfe9e523b2ba24a6c9de3c7
20:30:10 | Compiling packages: bosh_deployment_resource/56c0e245445bab5097dd6725fd99e98fa638b130 (00:00:04)
20:30:10 | Compiling packages: generated_signing_key/00b5b02050bc6588cdfe9e523b2ba24a6c9de3c7
20:30:11 | Compiling packages: generated_tsa_host_key/00b5b02050bc6588cdfe9e523b2ba24a6c9de3c7 (00:00:01)
20:30:11 | Compiling packages: concourse_version/33e3d7d5da79cadb3d6e09880eaa196df0ddea54
20:30:12 | Compiling packages: generated_signing_key/00b5b02050bc6588cdfe9e523b2ba24a6c9de3c7 (00:00:02)
20:30:12 | Compiling packages: golang/b1b2135422b3691f62b4dfd38d2f960759155697
20:30:12 | Compiling packages: docker_image_resource/b7ad85a4aa0e63bb89fc58682de209429a0db1ef (00:00:10)
20:30:12 | Compiling packages: pid_utils/a1f0590ea02d938b933a101c7438985721cd0ab4
20:30:12 | Compiling packages: concourse_version/33e3d7d5da79cadb3d6e09880eaa196df0ddea54 (00:00:01)
20:30:12 | Compiling packages: maximus/d37f10465e2adbfcb708773ab5f540f81bbaf884
20:30:13 | Compiling packages: pid_utils/a1f0590ea02d938b933a101c7438985721cd0ab4 (00:00:01)
20:30:13 | Compiling packages: guardian/d2b0ce380e1ddb59d71223894723055f1d31acd8
20:30:18 | Compiling packages: tar/f2ea61c537d8eb8cb2d691ce51e8516b28fa5bb7 (00:01:11)
20:30:18 | Compiling packages: runc-rootless/13f90a80fb14a002a60a9f7db518b9340422eb24
20:30:20 | Compiling packages: maximus/d37f10465e2adbfcb708773ab5f540f81bbaf884 (00:00:08)
20:30:20 | Compiling packages: runc/27c25b1fbc49b7cf5b4d35958ec9b1d290b9601a
20:30:34 | Compiling packages: golang/b1b2135422b3691f62b4dfd38d2f960759155697 (00:00:22)
20:30:34 | Compiling packages: baggageclaim/56f351a1e6394de4001a4d6e2af321f701b38719
20:30:35 | Compiling packages: runc-rootless/13f90a80fb14a002a60a9f7db518b9340422eb24 (00:00:17)
20:30:35 | Compiling packages: groundcrew/70115c7b6030f03e55f3b2e7404b1197442c574b
20:30:39 | Compiling packages: runc/27c25b1fbc49b7cf5b4d35958ec9b1d290b9601a (00:00:19)
20:30:39 | Compiling packages: resource_discovery/624b6ecbed549981d6aed42fcc99f8c35f3a0e59
20:30:41 | Compiling packages: guardian/d2b0ce380e1ddb59d71223894723055f1d31acd8 (00:00:28)
20:30:41 | Compiling packages: tsa/2dfac0651773697ea82974553960911c75925aba
20:30:44 | Compiling packages: baggageclaim/56f351a1e6394de4001a4d6e2af321f701b38719 (00:00:10)
20:30:44 | Compiling packages: fly/fa6033d0990d8d08458a3a04fb8dd7d9ccdf874b
20:30:45 | Compiling packages: groundcrew/70115c7b6030f03e55f3b2e7404b1197442c574b (00:00:10)
20:30:45 | Compiling packages: atc/3f8900840b6e83a458c92a38868822d7a98f8769
20:30:49 | Compiling packages: resource_discovery/624b6ecbed549981d6aed42fcc99f8c35f3a0e59 (00:00:10)
20:30:53 | Compiling packages: tsa/2dfac0651773697ea82974553960911c75925aba (00:00:12)
20:30:55 | Compiling packages: fly/fa6033d0990d8d08458a3a04fb8dd7d9ccdf874b (00:00:11)
20:31:02 | Compiling packages: atc/3f8900840b6e83a458c92a38868822d7a98f8769 (00:00:17)
20:32:56 | Compiling packages: postgresql_9.3/f8b7a26f105b921843fa5565818b50cc45ace6af (00:02:49)
20:32:56 | Creating missing vms: web/21ca23ff-57cf-4917-8f29-4c3cd65bca76 (0)
20:32:56 | Creating missing vms: worker/f2b9d34c-8691-4744-9213-093af0b1c08a (0)
20:32:56 | Creating missing vms: db/7894b167-7d44-4d45-99ca-36e397213441 (0)
20:33:03 | Creating missing vms: web/21ca23ff-57cf-4917-8f29-4c3cd65bca76 (0) (00:00:07)
20:33:04 | Creating missing vms: db/7894b167-7d44-4d45-99ca-36e397213441 (0) (00:00:08)
20:33:04 | Creating missing vms: worker/f2b9d34c-8691-4744-9213-093af0b1c08a (0) (00:00:08)
20:33:04 | Updating instance web: web/21ca23ff-57cf-4917-8f29-4c3cd65bca76 (0) (canary)
20:33:04 | Updating instance db: db/7894b167-7d44-4d45-99ca-36e397213441 (0) (canary)
20:33:04 | Updating instance worker: worker/f2b9d34c-8691-4744-9213-093af0b1c08a (0) (canary)
20:33:23 | Updating instance db: db/7894b167-7d44-4d45-99ca-36e397213441 (0) (canary) (00:00:19)
20:33:24 | Updating instance web: web/21ca23ff-57cf-4917-8f29-4c3cd65bca76 (0) (canary) (00:00:20)
20:33:40 | Updating instance worker: worker/f2b9d34c-8691-4744-9213-093af0b1c08a (0) (canary) (00:00:36)
Started Sat Apr 22 20:29:07 UTC 2017
Finished Sat Apr 22 20:33:40 UTC 2017
Duration 00:04:33
Task 11 done
Succeeded
できた。簡単。
$ curl -v -k https://10.244.0.200
* Rebuilt URL to: https://10.244.0.200/
* Trying 10.244.0.200...
* Connected to 10.244.0.200 (10.244.0.200) port 443 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 692 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification SKIPPED
* server certificate status verification SKIPPED
* common name: 10.244.0.200 (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: C=USA,O=Cloud Foundry,CN=10.244.0.200
* start date: Sat, 22 Apr 2017 20:39:36 GMT
* expire date: Sun, 22 Apr 2018 20:39:36 GMT
* issuer: C=USA,O=Cloud Foundry,CN=ca
* compression: NULL
* ALPN, server did not agree to a protocol
> GET / HTTP/1.1
> Host: 10.244.0.200
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Xss-Protection: 1; mode=block
< Date: Sat, 22 Apr 2017 21:08:10 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
<
<!DOCTYPE html>
<html lang="en">
<head>
<title>Concourse</title>
<link href="/public/main.css?id=a7cb3ee758b221a7b065e564185414b8" media="all" rel="stylesheet" type="text/css" />
<link id="favicon" rel="icon" type="image/png" href="/public/images/favicon.png?id=ac9c34f90047b8c51d980fd10cb8f23e" />
<meta name="theme-color" content="#000000" />
<script src="/public/index.js?id=932c553753eec4ef375e1c17f4b28c10"></script>
<script src="/public/d3.v355.min.js?id=5936da7688d010c60aaf8374f90fcc2b"></script>
<script src="/public/graph.js?id=3e2a3f561ec6c52216e80266ef0212d2"></script>
<script src="/public/jquery-2.1.1.min.js?id=e40ec2161fe7993196f23c8a07346306"></script>
<script src="/public/concourse.js?id=b4688d002041187f6c55f6e8699442ba"></script>
<script src="/public/elm.min.js?id=a832dfd7cb7039f375d056d9dd7ac46b"></script>
</head>
<body>
<div id="elm-app-embed"></div>
<script type="text/javascript">
var node = document.getElementById("elm-app-embed");
var app = Elm.Main.embed(node, {
turbulenceImgSrc: "/public/images/buckleup.svg?id=15ff39e468832758192d171ecc60c86d",
csrfToken: ""
});
app.ports.renderPipeline.subscribe(function (values) {
setTimeout(function(){
foundSvg = d3.select(".pipeline-graph");
var svg = createPipelineSvg(foundSvg)
if (svg.node() != null) {
var jobs = values[0];
var resources = values[1];
draw(svg, jobs, resources, app.ports.newUrl);
}
}, 0)
})
var teamNameRegExp = /\/teams\/([^/]+)\/.*/;
app.ports.requestLoginRedirect.subscribe(function (message) {
setTimeout(function(){
var path = document.location.pathname;
var query = document.location.search;
if(path.charAt(0) == "/") {
var redirect = encodeURIComponent(path + query );
var matches = path.match(teamNameRegExp);
var loginUrl;
if(matches) {
var teamName = matches[1];
loginUrl = "/teams/" + teamName + "/login?redirect="+ redirect;
} else {
loginUrl = "/login?redirect="+ redirect;
}
app.ports.newUrl.send(loginUrl);
}
}, 0)
});
app.ports.setTitle.subscribe(function(title) {
setTimeout(function(){
document.title = title + "Concourse";
}, 0)
});
var storageKey = "csrf_token";
app.ports.saveToken.subscribe(function(value) {
localStorage.setItem(storageKey, value);
});
app.ports.loadToken.subscribe(function() {
app.ports.tokenReceived.send(localStorage.getItem(storageKey));
});
</script>
</body>
</html>
* Connection #0 to host 10.244.0.200 left intact
HA Proxyで外部からアクセス可能にする
作りたいのは次のような環境(再掲)。
前述のインストールではConcourseのATCでTLS Terminationを行っていたが、今回はHA Proxy側でTLS Terminationを行い、Concourseには80番(HTTP)で転送する。
自己証明書の生成はBOSH v2 CLIで簡単にできる。
まずはConsourseにExternal IPを設定する。ここではconcourse-192-168-11-210.sslip.io (DNSで正引きすると192.168.11.210を返すIPアドレス)を使用する。
concourse.ymlの次の2箇所を修正。
- name: atc
release: concourse
properties:
external_url: &external_url https://((external_ip)) # ここを変更
basic_auth_username: admin
basic_auth_password: ((ui_password))
と
- name: atc_ssl
type: certificate
options:
ca: default_ca
common_name: ((external_ip)) # ここを変更
alternative_names: [((external_ip))] # ここを変更
次にcreds-concourse.ymlに先ほど生成した証明書が既に設定されているため、再生成するために、creds-concourse.ymlからatc_ssl.certificateとatc_ssl.private_key、atc_ssl.caを削除する。
削除したら、
bosh2 interpolate concourse.yml --vars-store ./creds-concourse.yml -v internal_ip=10.244.0.200 -v external_ip=concourse-192-168-11-210.sslip.io
で再度証明書を生成。
生成された証明書(atc_ssl.certificate)と秘密鍵(atc_ssl.private_key)はそれぞれ
bosh2 interpolate --path=/atc_ssl/certificate creds-concourse.yml
bosh2 interpolate --path=/atc_ssl/private_key creds-concourse.yml
で取得できる。これらを順番に並べたserver.pemとファイルを作成して/etc/haproxy/server.pemにおく。（空行を入れないように）
Concourse側はもうTLSの設定は不要なので、atc jobのpropertiesからtls_cert、tls_key、tls_bind_portを削除。
jobs:
- name: atc
release: concourse
properties:
external_url: &external_url https://((external_ip))
basic_auth_username: admin
basic_auth_password: ((ui_password))
# tls_cert: ((atc_ssl.certificate)) # コメントアウトまたは削除
# tls_key: ((atc_ssl.private_key)) # コメントアウトまたは削除
bind_port: 80
# tls_bind_port: 443 # コメントアウトまたは削除
postgresql_database: &atc_db atc
これでexternal_ipも設定してBOSHにデプロイ。
bosh2 deploy -d concourse --vars-store ./creds-concourse.yml -v internal_ip=10.244.0.200 -v external_ip=concourse-192-168-11-210.sslip.io concourse.yml
デプロイが完了すれば80番でATCにアクセスできることを確認。
curl 10.244.0.200
最後にHA Proxyの設定。/etc/haproxy/haproxy.cfgに次の設定を追加。
今後、Concourse以外にもBOSH管理でHTTPSを受け付けるものがあると想定して、Hostヘッダーがconcourse-192-168-11-210.sslip.ioのものだけをConcourseの80番に転送する。
frontend https_frontend
bind *:443 ssl crt /etc/haproxy/server.pem
reqadd X-Forwarded-Proto:\ https
acl host_concourse_sslip hdr(host) concourse-192-168-11-210.sslip.io
use_backend concourse_backend if host_concourse_sslip
backend concourse_backend
balance roundrobin
server concourse 10.244.0.200:80
BOSHでHTTPSをConcourseにしか使わない場合はdefault_backendでもOK。
frontend https_frontend
bind *:443 ssl crt /etc/haproxy/server.pem
reqadd X-Forwarded-Proto:\ https
default_backend concourse_backend
backend concourse_backend
balance roundrobin
server concourse 10.244.0.200:80
あとは
sudo service haproxy restart
これで
https://concourse-192-168-11-210.sslip.ioにアクセスできる。
curl -k https://concourse-192-168-11-210.sslip.io
Hostヘッダーのチェックをしていれば、他のホスト名だとNGになる。
$ curl -k https://foo-192-168-11-210.sslip.io
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
これでセットアップ環境。
HA Proxy版のmanifestはこちらにpushしておいた。
Happy BOSH Life🐚を送れそう。