Devops on inherent site

Ansible, Terraform and Proxmox

Fri, 24 Jun 2022 23:24:57 +0300

In the continuing quest to improve my homelab I’ve written a few ansible playbooks as well as terraform configuration to automate a lot of tasks. The initial setup of proxmox is handled by ansible, the provisioning of virtual machines is handled by terraform and the installation of k3s on the VMs is done by ansible.

Proxmox Setup with Ansible

Before getting to use all the shiny devops with a selfhosted solution, we need to do a bit of setup first. This isn’t perfect since it’s just ansible running a bash script but it gets the job done. The other issue,, is that a lot of cloud images don’t include qemu-guest-agent and other packages by default. I have a few ideas how to solve that (check out this TODO.md for more info) but haven’t tested out anything yet. Since qemu-guest-agent and things like nfs-common are required on most VMs, a lot of redundant package installation happens but such is life for now. With that out of the way, let’s start with how I configure proxmox.

Host

While proxmox has a lot of things ready to go out of the box, a bit of configuration is always required and even more so in this case. The goal of the following steps is to turn our hosts into capable, cloud-like, targets for provisioning resources using terraform.

Repository

Initially the enterprise repository which we don’t have a license to access is enabled. To fix this, I delete the file in /etc/apt/sources.list.d that contains it and add a file with the no-subscription repository. This might not be necessary since the lae.proxmox ansible role can do essentially the same thing and many more. Here are the ansible tasks:

    - name: remove enterprise repo
      file:
        path: /etc/apt/sources.list.d/pve-enterprise.list
        state: absent

    - name: add no-subscription repo
      copy:
        src: pve-no-sub.list
        dest: /etc/apt/sources.list.d/pve-no-subscription.list
        mode: '0644'
        owner: root
        group: root

Template VM

The terraform provider for proxmox can work with cloud-init which is a great vendor-agnostic way of configuring the basics (username, password, IP addressing, DNS, SSH keys) in a virtualized environment. This is achieved by having a virtual disk or virtual cd-rom that mounts inside the VM and provides this information to the guest.

VM Image

Since software is not always compatible with the latest version of a linux distribution, in this case debian, I keep the last two versions around. The first step is to download the images:

    - name: get debian 10 cloud image
      get_url:
        url: https://cloud.debian.org/images/cloud/buster/20211011-792/debian-10-generic-amd64-20211011-792.qcow2
        dest: /root/debian-10-openstack-amd64.qcow2
        checksum: sha512:f3dac13104b4e28eb62c46cbbb1e30fc9c792834500f9101d477c19c258c31ff04850933ee0b3e63236eca38c854447d95a0ab45163c4a3fccf05f9dd95601a5
        mode: '0644'
        owner: root
        group: root

    - name: get debian 11 cloud image
      get_url:
        url: https://cloud.debian.org/images/cloud/bullseye/20220121-894/debian-11-generic-amd64-20220121-894.qcow2
        dest: /root/debian-11-openstack-amd64.qcow2
        checksum: sha512:0948dc56b4834a7755e4eae7d5532e138b90484a949161ffe9fc6894c7a14b1bd32ebf96fa3f3d03d498fe7ee125f8014f31bfab5825915a93de9330df814f7b
        mode: '0644'
        owner: root
        group: root

Template Script

As mentioned in the beginning I’d like to automate this with ansible but this is what I have for now:

if [ -z "$1" ]; then
    VMID=9001
else
    VMID="$1"
fi

if [ -z "$2" ]; then
    VMNAME="debian-tmpl"
else
    VMNAME="$2"
fi

VMEXISTS="$(qm list | grep $VMID)"

if [ "$VMEXISTS" ]; then
    echo "VM with ${VMID} already exists"
else
    qm create "${VMID}" -name "${VMNAME}" -memory 1024 -net0 virtio,bridge=vmbr0 -cores 1 -sockets 1 -cpu cputype=kvm64 -description "debbie image" -kvm 1 -numa 1
    qm importdisk "${VMID}" debian-10-openstack-amd64.qcow2 local-zfs
    qm set "${VMID}" -scsihw virtio-scsi-pci -virtio0 local-zfs:vm-"${VMID}"-disk-0
    qm set "${VMID}" -serial0 socket
    qm set "${VMID}" -boot c -bootdisk virtio0
#   qm set "${VMID}" -agent 1 #disabled since VM does not initially have qga installed
    qm set "${VMID}" -hotplug disk,network,usb,memory,cpu
    qm set "${VMID}" -vcpus 1
    qm set "${VMID}" -vga qxl
    qm set "${VMID}" -name "${VMNAME}"
    qm set "${VMID}" -ide2 local-zfs:cloudinit
    qm template "${VMID}"
fi

This works well enough for first-time setup and with a few additions could become idempotent. The above script and its debian 11 equivalent are copied over to the proxmox hosts and executed by the following ansible tasks:

    - name: get debian 10 template vm script
      copy:
        src: setup-debian-10-cloudinit.sh
        dest: /root/setup-debian-10-template.sh
        mode: '0755'
        owner: root
        group: root

    - name: run debian 10 template vm script
      shell:
        cmd: /root/setup-debian-10-template.sh

    - name: get debian 11 template vm script
      copy:
        src: setup-debian-11-cloudinit.sh
        dest: /root/setup-debian-11-template.sh
        mode: '0755'
        owner: root
        group: root

    - name: run debian 11 template vm script
      shell:
        cmd: /root/setup-debian-11-template.sh

Terraform Setup

With proxmox ready, let’s move on to the terraform setup. If you haven’t used it before, terraform is a declarative way of describing our infrastructure. It can manage virtual machines, DNS records, object storage buckets, kubernetes resources and many more things with the appropriate providers.

Provider

Providers in terraform are plugins that enable terraform to manage resources that it doesn’t support out of the box. In this case, we want to manage proxmox virtual machines and containers but it could be other things like a router running pfsense or vyos. In previous versions, the provider would need to be installed manually but nowadays we only need to create a main.tf with the provider listed and upon running terraform init, the provider will be downloaded automatically.

Basic `main.tf`

Let’s create that file then. Open your favorite editor, paste the following and replace the IP as well as the password with the ones applicable to your setup:

terraform {
	required_providers {
		proxmox = {
			source = "Telmate/proxmox"
			version = ">=2.9.5"
		}
	}
}

provider "proxmox" {
	alias = "pve"
	pm_tls_insecure = true
	pm_api_url = "https://192.168.70.1:8006/api2/json" #Replace IP
	pm_password = "password123" #Replace password
	pm_user = "root@pam"
}

Save the file and then run terraform init. You should see the following output (the version might be different):

Initializing the backend...

Initializing provider plugins...
- Finding telmate/proxmox versions matching ">= 2.9.5"...
- Installing telmate/proxmox v2.9.10...
- Installed telmate/proxmox v2.9.10 (self-signed, key ID A9EBBE091B35AFCE)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

And ta-da! We’ve installed the provider and are ready to start creating resources.

Creating a test VM

The above file won’t do anything so we’ll declare a virtual machine and then tell terraform to create it. The example below is specific to my setup so you probably want to modify it before using but here it is:

terraform {
	required_providers {
		proxmox = {
			source = "Telmate/proxmox"
			version = ">=2.9.5"
		}
	}
}

provider "proxmox" {
	alias = "pve"
	pm_tls_insecure = true
	pm_api_url = "https://192.168.70.1:8006/api2/json" #Replace IP
	pm_password = "password123" #Replace password
	pm_user = "root@pam"
}

resource "proxmox_vm_qemu" "proxmox_test_vm" {
	provider = proxmox.pve
	name = "deb-10-test"
	target_node = "pve"

	/* change to debian-templ for 11 */
	clone = "debian-tmpl"
	os_type = "cloud-init"
	cores = 2
	sockets = 1
	cpu = "host"
	memory = 2048
	scsihw = "virtio-scsi-pci"
	bootdisk = "virtio0"
	agent = 0

	disk {
		size = "30G"
		type = "virtio"
		storage = "local-zfs"
	}

	network {
		model = "virtio"
		bridge = "vmbr0"
	}

	ipconfig0 = "ip=192.168.50.25/16,gw=192.168.0.1"

	/*workstation SSH key*/
	sshkeys = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5jzKi37jm3517bqThbw+7LR/GXm3qC6Az5F+ZUa36vYM7Ygk2K5bWcFIL2YUCrkL5jfSsvoowONjCAxyuoyxtW4MJxnQLyq4u4yDsRC7YvBPAKZUYaHwnbkCfDs5a75dEFOoDxCA0DY2GrhqzBndaTcCfl0fZ4vN+9LcKOb1dSKiHeHvsh35YNtwntbL21meo+hiycUEgGwNe9/4kxKpdGTr7HvbeX2Fjm/UZBZIJKVcGop/3gCHXYnKH+OY5zc8cmt9Jg4CIwEqrSKeOX0bE8LSPRpVRXH4v8OcMaMei/HQejlH8NBwybEdJ4mhl8vHaFEjDbIWoOujmiRQF2263 angle@puddle"

	/*required otherwise the state always appears modified*/
	lifecycle {
		ignore_changes = [
			cipassword,
			network,
			desc,
		]
	}
}

After saving the file, we can run terraform plan -out create-test-vm.plan to calculate the changes required and save them to a file. Following that, we can run terraform apply create-test-vm.plan to apply the changes to our infrastructure. Some output will appear while the template is cloned, the VM settings are adjusted and the cloud-init configuration is put into the drive. In the end, you should have a brand new VM created by terraform!

Creating VMs for k3s

The above example is realistic but what if we need 5 similar VMs that could be used as kubernetes nodes? Conveniently, terraform provides a count variable that we can use to slightly differ settings where needed. Here is the resource that provisions the virtual machines that I run a k3s cluster on:

resource "proxmox_vm_qemu" "proxmox_vm_k3s" {
	provider = proxmox.pve
	count = 5
	name = "deb-k3s-${count.index + 1}"
	target_node = "pve"

	clone = "debian-tmpl"
	os_type = "cloud-init"
	cores = 2
	sockets = 1
	cpu = "host"
	memory = 3084
	scsihw = "virtio-scsi-pci"
	bootdisk = "virtio0"
	agent = 0
	onboot = true

	disk {
		size = "30G"
		type = "virtio"
		storage = "local-zfs"
	}

	network {
		model = "virtio"
		bridge = "vmbr0"
	}

	ipconfig0 = "ip=192.168.15.5${count.index + 1}/16,gw=192.168.0.1"

	sshkeys = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5jzKi37jm3517bqThbw+7LR/GXm3qC6Az5F+ZUa36vYM7Ygk2K5bWcFIL2YUCrkL5jfSsvoowONjCAxyuoyxtW4MJxnQLyq4u4yDsRC7YvBPAKZUYaHwnbkCfDs5a75dEFOoDxCA0DY2GrhqzBndaTcCfl0fZ4vN+9LcKOb1dSKiHeHvsh35YNtwntbL21meo+hiycUEgGwNe9/4kxKpdGTr7HvbeX2Fjm/UZBZIJKVcGop/3gCHXYnKH+OY5zc8cmt9Jg4CIwEqrSKeOX0bE8LSPRpVRXH4v8OcMaMei/HQejlH8NBwybEdJ4mhl8vHaFEjDbIWoOujmiRQF2263 angle@puddle"

	lifecycle {
		ignore_changes = [
			cipassword,
			network,
			desc,
		]
	}
}

There aren’t many changes from the previous example, the name and the IP address use the index in order to avoid 5 VMs having the same name and IP address and that’s it!

Conclusion

Hopefully this gives you inspiration for some awesome things you can do to automate your infrastructure and make working in a virtualized environment easier. Thank you for reading.

Flux Update

Wed, 25 Aug 2021 00:44:34 +0300

I wrote about my journey of choosing a continuous deployment tool and why I ultimately ended up using flux for my homelab in this post. It took quite a bit of reasearch and if you’re in a similar position as I was it might prove to be helpful.

Long(ish)-term experience

I’ve been using it for about 5 months now, the first post was written about one month of use. At that point I had been tinkering with it and attempting to get all the stuff I wanted to be deployed by it. Some things were simple, others not so straight-forward.

Successes

Deploying raw manifest applications

Deploying raw kubernetes manifests was really easy. Since I’m somewhat on the fence about Helm, I prefer using normal kubernetes manifests where possible. This meant that for my use case mostly everything was smooth sailing after figuring out how to generate a kustomization yaml using the command-line tool.

Command-line tool

Speaking of that, flux has a command-line tool to interact with the controllers running inside the cluster. It’s also the way you can initialize a repository to be used as the place for flux files to be stored. It can also be used to install flux to the cluster if the repository already exists.

Upgrading

When a new version of flux comes out, you can upgrade and it will carry over to the cluster. It’s really easy, it just upgrades its own files so when those are committed it will pull in the new version of the controllers.

Trouble

Aggregation yaml

For starters here is a minor one that took way too long to diagnose. I forget exactly what this is called but essentially it is a yaml file says run deal with the listed resources. Imagine the following directory contents:

drwxrwxr-x 4 angle angle 4,0K Αυγ  25 01:23 .
drwxrwxr-x 5 angle angle 4,0K Ιουλ 21 03:42 ..
drwxrwxr-x 2 angle angle 4,0K Αυγ  13 15:55 helmrepos
-rw-rw-r-- 1 angle angle  219 Ιουλ 21 03:20 infra-source.yml
-rw-rw-r-- 1 angle angle  126 Αυγ  12 11:45 kustomization.yaml
drwxrwxr-x 2 angle angle 4,0K Αυγ  13 15:56 storage

The kustomization.yaml file is as follows:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - infra-source.yml
  - helmrepos
  - storage

So it’s a way to more easily include stuff. The issue I had is that I named the file kustomization.yml instead of kustomization.yaml which for some reason isn’t supported. I believe this is a bug and not intended behavior but it was still frustrating to find and fix.

Logs

Now on to more substantial issues. While troubleshooting I got a bit frustrated with how flux displays errors. The messages are fairly non-descript and there isn’t a lot of guidance for how to fix them.

Conclusion

Despite a couple rough edges, flux has been working incredibly well. Even before I got it fully functioning, it was amazing to be able to just install it to a newly creted cluster and have all my applications running just like that. It’s almost magical and I’ve been enjoying it quite a lot. If you aren’t using a continuous deployment tool, feel free to give flux a try or read the previous post to learn about some other options

Docker Multi-Binary Images

Sun, 01 Aug 2021 17:43:30 +0300

Problems in deploying distributed applications

If you read my background in the front page, you know I’m not a prolific programmer by any definition. This isn’t going to be a gigantic revalation and if you’ve worked with deploying your own applications to kubernetes you’ll already know most of the following things. With that said, what are the problems with deploying a distributed application? I was tinkering with gifinator and while writing the kubernetes manifests I realized there was a problem. A few bugs required multi-service re-deployments. Let’s say there was something affecting the render service and the gifcreator worker service. I could rebuild 2 new docker images, push them to the local image repository, update the version in their respective manifests and apply the changes. The issue is that after doing it once or twice it became really annoying when wanting to roll back to check the previous behavior. The more this cycle went on, I had to keep track of a set of 4 different versions which quickly became unmanageable.

Lightbulb moment

That’s when the obvious dawned on me, the upstream kubernetes manifests all used the same image somehow. I quickly opened them up and checked to see they were redefining the container’s CMD directive! In fact the actual Dockerfile did not even have a CMD nor ENTRYPOINT defined. The code was built, the binaries were copied to the second stage and the final image was shipped. Using this method the final image is as small as possible without needing 4 separate ones.

How do

It’s pretty easy to do this so let’s take a look at the Dockerfile as well as the kubernetes manifests.

Dockerfile

This specific Dockerfile is available on gitlab for those interested but I’ll paste it here for ease:

# build stage
FROM golang:1.16 as build

ENV CGO_ENABLED 0
ENV GOOS linux
ENV GOARCH amd64
ENV GO111MODULE on

WORKDIR /go/src/gifinator
COPY . .

WORKDIR /go/src/gifinator/render
RUN go get -v
RUN go vet -v
RUN go install -v

WORKDIR /go/src/gifinator/gifcreator
RUN go get -v
RUN go vet -v
RUN go install -v

WORKDIR /go/src/gifinator/frontend
RUN go get -v
RUN go vet -v
RUN make installwithvars

# run stage
FROM busybox as run
RUN mkdir /tmp/objcache
RUN mkdir /tmp/scene
COPY --from=build /go/bin/render /render
COPY --from=build /go/bin/gifcreator /gifcreator
COPY --from=build /go/bin/frontend /frontend
COPY ./gifcreator/scene /tmp/scene
COPY ./frontend/templates /templates
ENV FRONTEND_TEMPLATES_DIR=/templates

First up we set some basic environment variables and copy the code into the building environment. Then one by one the services are compiled and installed into the build stage image. Following that, in the run stage the needed directories are created, the binaries are copied over, assets are also copied over and a final environment variable is set. There is no CMD or ENTRYPOINT as you can see, this will probably result in the inability to just docker run -p externalport:internalport imagename but the tradeoff is well worth it.

Kubernetes deployment manifests

With the Dockerfle out of the way, let’s look at the way these images are deployed. The primary way I run containers is through kubernetes so we’ll be looking at kubernetes manifests. Same as the Dockerfile you can find the source for these manifests on gitlab, I’ll be linking each one. kube/manifests/gifinator/render/deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: render
  namespace: gifinator
  labels:
    app: render
spec:
  replicas: 1
  selector:
    matchLabels:
      app: render
  template:
    metadata:
      labels:
        app: render
    spec:
      containers:
      - name: render
        image: inherently/gifinator:0.0.5
        command: ["/render"]
#the rest of the file is omitted

kube/manifests/gifinator/gifcreator-server/deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gifcreator-server
  namespace: gifinator
  labels:
    app: gifcreator-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gifcreator-server
  template:
    metadata:
      labels:
        app: gifcreator-server
    spec:
      containers:
      - name: gifcreator-server
        image: inherently/gifinator:0.0.5
        command: ["/gifcreator"]
#the rest of the file is omitted

kube/manifests/gifinator/gifcreator-worker/deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gifcreator-worker
  namespace: gifinator
  labels:
    app: gifcreator-worker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gifcreator-worker
  template:
    metadata:
      labels:
        app: gifcreator-worker
    spec:
      containers:
      - name: gifcreator-worker
        image: inherently/gifinator:0.0.5
        command: ["/gifcreator", "-worker"]
#the rest of the file is omitted

kube/manifests/gifinator/frontend/deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: gifinator
  labels:
    app: frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        image: inherently/gifinator:0.0.5
        command: ["/frontend"]
#the rest of the file is omitted

Here are 4 different services, all easily using the same image. Just by redifining the command it’s easy to simplify the development process by a quite a lot. It’s worth noting that I’ve only shown the deployment manifests since the other ones don’t have anything to do with the container image used.

Conclusion

This is a short post mostly made up of code snippets but I hope you learned something and can improve your workflow with distributed microservice applications.

Droning On

Sat, 01 May 2021 18:54:59 +0300

I described previously how the deployment of applications is automated in my homelab but didn’t touch upon how I test my own code. The topic came up when I started trying to simulate a development pipeline who’s first step after committing code is to be run through the continuous integration system. What should that system be though? In this post, I’ll take you through the things I tried first and then why I went with drone in the end.

Choose a Fighter

My quest is to ultimately end up with the skills to set up kubernetes so that it can be deployed to everything from a DIY NAS and a stack of NUCs to a warehouse full of servers backed by distributed storage. I’m obviously more towards the first camp so that’s where my journey is at. With that said, my goal is to have software small and lightweight enough that it can run on the former setup easily so that there can be multiple instances of it in the latter form. The current mental model I have is developer organization with not much more than 3 NUCs + 1 NAS per team without losing the ability for continuous integration and continuous deployment. Therefore the landscape needs to be explored and the obvious titan of the old world is jenkins with many newcomers eager to take its place. The obvious caveats still apply, it must be open-source, able to be self-hosted, support for gitea is needed and being able to run well on kubernetes is a foregone conclusion.

Jenkins

It should need no introduction, most developers are familiar with the trusty crusty old butler software. Jenkins is probably one of the most prominent pieces of CI/CD software that was pushed to the forefront when the push for more automation and increased code quality really happened. Sadly there are a few reasons that annoyed me from the get-go. First, the unlocking procedure requires you to kubectl logs <podname> in order to find the unlock key which I understand but nonetheless find annoying. Second, the Jenkinsfile has its own domain-specific language (DSL for short) that I find unappealing even though you won’t catch me praising YAML’s lack of curly braces. I begrudgingly went though it and wrote a Jenkinsfile to give it a fair chance but it wasn’t a nice experience. Next, a single instance uses too much ram to the point that it can barely fit in my k3s VMs which use 2.5GB of ram each together with other software. This is an issue because I like running at least 2 instances of most applications to be sure that everything works correctly when scaling up. Last but not least, it’s a 2-in-1 solution which I don’t find appealing as far as architecture and since I was planning to find a standalone piece of software to do CI and another one to handle CD, it was a hard sell from the start. As a redeeming quality it has an official kubernetes pipeline plugin which is nice so they’ve at least put thought into this use case. Ultimately it wasn’t the right fit for me so it was time to move on.

Drone

While pondering this I remembered an old post from one of the blogs I read, specifically this one. I said what the hell, why not and started looking its website. Drone is a very nice and simple continuous integration tool. From the start it seemed like it would be a good pick but you never know until you try. I wrote some kubernetes manifests, got angry until I got them to work and a half hour later it was hooked up to my local gitea instance and ready to use. I added a .drone.yml to one of my repos, pushed the change, activated the repo from the drone dashboard, made another change to the source code, pushed it and waited to see what happened. Instantly drone received the webhook event and got to work. The dashboard is pretty simple (and sadly lacks a dark mode) so you just hit the thing you want to look at, click on the build and see it happen. It supports different things running at the same time like databases if you want your tests to include that which can be run either as what drone calls services aka run in the background from the beginning or if you want an ordered dependency you can use detached steps which is especially useful in microservices. My only real complaint aside from the lack of dark mode is the inability to trigger a build manually without pushing but that’s only an issue when you first activate a repository inside drone’s dashboard. All in all, it has been a joy to use, most of the projects I am currently developing have a .drone.yml and get built at least 3 times a day when gitea mirrors the gitlab or github repo they’re mainly hosted at.

Conclusion

This wasn’t as big of a journey as finding a continuous development tool because really I only found the two aforementioned options that had the basics I was looking for and one of them didn’t really fit the bill. I’m not huge into software development, it’s mostly a hobby for me as might be obvious by now, since my main concern is infrastructure-related stuff. With that said it’s useful to have the skills to build an application that can communicate with the kubernets api to automate something or write an ansible module and most of all it allows me to see the developer perspective. I haven’t started looking into SOPS or renovate bot or the k3s upgrade controller or velero but I hope to get the chance to do so in the future. Thank you for reading, I hope you enjoyed it and maybe learned something.

Fluxing My Cluster

Sat, 17 Apr 2021 17:24:22 +0300

It’s no secret that I’m a fan of automation and making life easier (even if I made it harder in the first place). One of the issues I’ve been having in my homelab is dealing with deploying stuff to kubernetes. Initially I wanted to just add a plugin to drone and be done with it. However, that didn’t really pan out which ended up being to my benefit since I discovered flux.

The problem

The cluster in its current state is a little sad because I end up doing too many things manually. Whenever something changes it’s time to bring out ye old kubectl apply -f in a local clone of the repo and that doesn’t spark joy. This was obviously bad form and couldn’t continue. What you also have to keep in mind is that I’ve chosen a 3rd-party storage plugin, democratic-csi and assume it’s the default storageclass in the cluster. This means that on a new cluster it’s required that before anything else, that one is added as a storageclass and set as default and local-path is removed from being a default. After that, the applications can be deployed which includes a mix of stuff I’ve written for the purposes of testing as well as 3rd-party software. Now that the stage has been set, let’s see what I tried

Solutions

Drone CI plugin

Initially the idea was that since I’ve alrady solved this for code using drone for CI (this will be discussed in a later post), I could just use a plugin and be done. However, my experience with writing helm charts is limited and the only official plugin for deploying to kubernetes is for helm charts, not kubernetes manifests. Furthermore, as I mentioned previously, not all software deployed on the cluster is something that I have written so even if there was an official kubernetes manifest plugin it wouldn’t cover 3rd-party software. At this point, I was thinking of finding some all-inclusive CI/CD software that would cover everything I wanted to do. It took no more than 10 seconds of thinking to realize that I was looking for non-modular software and discard that idea.

Standalone Continuous Delivery

This was the obvious choice but the software in this category is plentiful. The main competitors were flux, argo and tekton. Their designs differ quite a bit so there is going to be quite a bit of opinion in the following analysis so just keep that in mind. The few things they had in common were that they were all advertised as cloud-native with support for kubernetes, had a cli and were written in Go.

ArgoCD

I started with argo because one of the people that wrote a really good blog post on using kubernetes at home mentioned that they use it so I thought I’d try it first. The installation process required a massive yaml file that you just curl | kubectl apply -f which I wasn’t a big fan of but alright whatever. It also had a weird unlock procedure like jenkins where you have to find a token generated at runtime to unlock it but that’s okay, just an one-time setup thing. After spending a couple evenings with it, I wasn’t satisfied with the experience. The docs weren’t really good at explaining all the argo custom resources and getting a basic single-pod application running took me more than 2 hours. It was doable but this was going to be something I had to do for every application running on my cluster and the prospect didn’t seem appealing. I’m sure with more time I could maybe become more familiar and eventually warm up to it however first impressions were bad so I decided to move on.

Tekton

Tekton’s marketing was interesting so I took a look at it next. It seemed to be easier to set up and a couple random blog posts seemed to praise it so I downloaded another massive installation yaml which the guide told me to just kubectl apply -f which is okay I guess. I spent a few hours with it and it became clear you basically had to use it both for CI and CD so I put it to the side and moved on.

Flux

So why did I mention this one first but try it last? A few reasons. While researching, the v1 -> v2 development effort was going on and v2 didn’t have a lot of features but v1 was being phased out so I didn’t know if I should spend time learning v1 while v2 was coming along or if it’s worth jumping into alpha/beta stage software to avoid using legacy versions. By the time I did get around to it though v2 was clearly the way forward. Not only was it officially endorsed but among people that self-host, it was already the way to go 6-12 months ago. The docs were pretty clear, the cli was cross-platform and could be installed on a Raspberry Pi (something that argo at the time didn’t have available) and the examples worked. In an evening I had learned basically everything I needed to in order to just drop it in, generate the flux CRDs for my existing kubernetes manifests as well as helm charts and be up and running. The only issue I had, probably just a pet peeve of mine, was that the repo it creates on github is private by default and my goal is to have everything public and also secure (not there yet) but whatever. Flux v2 also mentioned as the gitops toolkit is made up of several different parts that have a specific purpose which is something I certainly appreciate. All in all, I think I’m going to stick with flux but you never know how homelab things will end up.

Conclusion

Flux is a really great project and I’ve enjoyed using it so far. The problem of continuous delivery has been solved for me and I recommend checking out if you’re in a similar position. Next up, I’m going to look at renovate bot for automating image updates, SOPS for secret management (which flux supports and an upgrade controller) to further help automate operations. Thank you for reading, I hope you enjoyed it and maybe learned something.

Devops on inherent site

Ansible, Terraform and Proxmox

Proxmox Setup with Ansible

Host

Repository

Template VM

VM Image

Template Script

Terraform Setup

Provider

Basic main.tf

Creating a test VM

Creating VMs for k3s

Conclusion

Flux Update

Long(ish)-term experience

Successes

Deploying raw manifest applications

Command-line tool

Upgrading

Trouble

Aggregation yaml

Logs

Conclusion

Docker Multi-Binary Images

Problems in deploying distributed applications

Lightbulb moment

How do

Dockerfile

Kubernetes deployment manifests

Conclusion

Droning On

Choose a Fighter

Jenkins

Drone

Conclusion

Fluxing My Cluster

The problem

Solutions

Drone CI plugin

Standalone Continuous Delivery

ArgoCD

Tekton

Flux

Conclusion

Basic `main.tf`