Speeding up the provisioning process by installing things ahead of time
As a frequent user of Terraform, I have a variety of modules tailored for different purposes, many of which rely on CentOS with a Docker install script. However, while demoing a setup to a co-worker, we encountered a significant delay — about seven minutes — for a Terraform apply to complete with an install script. Determined to improve this, I dedicated my Sunday to exploring Packer.
Given my extensive use of Terraform, delving into Packer seemed like a natural next step. Despite finding the initial documentation somewhat confusing, expecting it to be similar to Terraform, I quickly discovered that Packer is both straightforward and powerful.
My primary use case involves installing specific versions of Docker CE and Kubeadm, as most of my Terraform projects require a Docker setup. Running the full installation script usually takes around five minutes, not counting additional configuration steps or host-specific installs. While I could use the pre-configured Docker image on DigitalOcean, I prefer the flexibility and control of installing specific Docker CE versions.
Typically, my process involves provisioning a set of master machines (such as Docker Swarm mode masters, etcd, or Kubernetes API servers), followed by a series of worker nodes. This entire setup often results in a waiting period of around ten minutes, in addition to the time required for other installations and configurations. By integrating Packer into my workflow, I aim to significantly reduce these delays, enhancing efficiency and productivity.
I figured I’d start out simple and take the DigitalOcean builder example from Packer.
{
"type": "digitalocean",
"api_token": "YOUR API KEY",
"image": "ubuntu-14-04-x64",
"region": "nyc3",
"size": "512mb",
"ssh_username": "root"
}
And turn it into a packer config file:
{
"builders": [
{
"type": "digitalocean",
"api_token": "MY_TOKEN",
"image": "centos-7-x64",
"region": "ams3",
"size": "s-1vcpu-1gb",
"ssh_username": "root"
}
],
"provisioners": [
{
"type": "shell",
"script": "digitalocean/scripts/install-kubeadm.sh"
}
]
}
And the contents of digitalocean/scripts/install-kubeadm.sh:
#!/bin/bash
# install docker
yum update -y
yum install -y docker
systemctl enable docker && systemctl start docker
# get kubernetes repo
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
setenforce 0
# install kubernetes
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
Building the image turns out to be as simple as:
$ packer build docker-kubeadm.json
digitalocean output will be in this color.
==> digitalocean: Creating temporary ssh key for droplet...
==> digitalocean: Creating droplet...
==> digitalocean: Waiting for droplet to become active...
==> digitalocean: Waiting for SSH to become available...
==> digitalocean: Connected to SSH!
==> digitalocean: Provisioning with shell script: digitalocean/scripts/install-kubeadm.sh
digitalocean: Loaded plugins: fastestmirror
digitalocean: Determining fastest mirrors
digitalocean: * base: mirror.denit.net
digitalocean: * extras: mirror.nforce.com
digitalocean: * updates: centos.mirror.triple-it.nl
...
==> digitalocean: Gracefully shutting down droplet...
==> digitalocean: Creating snapshot: packer-1523190279
==> digitalocean: Waiting for snapshot to complete...
==> digitalocean: Destroying droplet...
==> digitalocean: Deleting temporary ssh key...
Build 'digitalocean' finished.
==> Builds finished. The artifacts of successful builds are:
--> digitalocean: A snapshot was created: 'packer-1523190279' (ID: 33289213) in regions ''
After successfully running the Packer build step, I can now see the available images using doctl:
$ doctl compute image ls
ID Name Type Distribution Slug Public Min Disk
33289213 packer-1523190279 snapshot CentOS false 25
I typically use a couple of different configurations: one with plain Docker CE, using one of the latest stable builds, and another with Docker alongside kubeadm.
By making a small modification to the standard Docker CE installation script, I created a shared script that efficiently installs Docker CE for both configurations:
#!/bin/bash
# Installing Docker CE On CentOS distributions
# https://docs.docker.com/install/linux/docker-ce/centos/#install-docker-ce
export VERSION=${VERSION:-18.03.0.ce-1.el7.centos}
yum update -y
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
yum install -y docker-ce-$VERSION
systemctl start docker
Now I just needed to add some configuration for creating an image for multiple docker versions. I started with Docker v17.12.0 and Docker v18.03.0.
{
"builders": [
{
"type": "digitalocean",
"api_token": "MY_TOKEN",
"image": "centos-7-x64",
"region": "ams3",
"size": "s-1vcpu-1gb",
"ssh_username": "root",
"snapshot_name": "docker-v17.12.0-ce",
"snapshot_regions": ["ams3"]
}
],
"provisioners": [
{
"type": "shell",
"environment_vars": ["VERSION=17.12.0.ce-1.el7.centos"],
"script": "digitalocean/scripts/install-docker-ce.sh"
}
]
}
When building with Packer for several different Docker versions, I ended up with the following images:
$ doctl compute image ls
ID Name Type Distribution Slug Public Min Disk
33289213 packer-1523190279 snapshot CentOS false 25
33289232 docker-v17.12.0-ce snapshot CentOS false 25
33289248 docker-v18.03.0-ce snapshot CentOS false 25
In Provisioning a Swarm mode cluster I talked about the Terraform module I use for creating a quick Docker swarm mode lab. I always either used the Docker image provided by DigitalOcean or ran an install script using user_data.
Instead I am now able to point towards the image I just created using Packer:
data "digitalocean_image" "docker_image" {
name = "docker-v17.12.0-ce"
}
module "swarm-cluster" {
source = "thojkooi/docker-swarm-mode/digitalocean"
version = "0.1.0"
total_managers = 1
total_workers = 1
domain = "do.example.com"
do_token = "${var.do_token}"
manager_ssh_keys = "${var.ssh_keys}"
worker_ssh_keys = "${var.ssh_keys}"
manager_os = "${data.digitalocean_image.docker_image.image}"
worker_os = "${data.digitalocean_image.docker_image.image}"
provision_user = "root"
manager_tags = ["${digitalocean_tag.cluster.id}", "${digitalocean_tag.manager.id}"]
worker_tags = ["${digitalocean_tag.cluster.id}", "${digitalocean_tag.worker.id}"]
}
Packer has proven to be a straightforward and effective tool that I will definitely be incorporating more into my workflow. The cost efficiency is notable as well; storing a snapshot on DigitalOcean costs only around $0.05 at the time of writing.
Considering the size of the images I’ve created, this amounts to approximately $0.30 a month. This small investment translates to a significant time savings of about five minutes every time I provision a new droplet. By using Packer for creating Kubernetes base images, I’ve streamlined my setup process, making it more efficient and cost-effective.
$ doctl compute snapshot ls
ID Name Created at Regions Resource ID Resource Type Min Disk Size Size
33289213 packer-1523190279 2018-04-08T12:29:23Z [ams3] 88805641 droplet 25 1.57 GiB
33289232 docker-v17.12.0-ce 2018-04-08T12:36:51Z [ams3] 88806114 droplet 25 1.34 GiB
33289248 docker-v18.03.0-ce 2018-04-08T12:45:46Z [ams3] 88806835 droplet 25 1.37 GiB