Flux Update

Wed, 25 Aug 2021 00:44:34 +0300

I wrote about my journey of choosing a continuous deployment tool and why I ultimately ended up using flux for my homelab in this post. It took quite a bit of reasearch and if you’re in a similar position as I was it might prove to be helpful.

Long(ish)-term experience

I’ve been using it for about 5 months now, the first post was written about one month of use. At that point I had been tinkering with it and attempting to get all the stuff I wanted to be deployed by it. Some things were simple, others not so straight-forward.

Successes

Deploying raw manifest applications

Deploying raw kubernetes manifests was really easy. Since I’m somewhat on the fence about Helm, I prefer using normal kubernetes manifests where possible. This meant that for my use case mostly everything was smooth sailing after figuring out how to generate a kustomization yaml using the command-line tool.

Command-line tool

Speaking of that, flux has a command-line tool to interact with the controllers running inside the cluster. It’s also the way you can initialize a repository to be used as the place for flux files to be stored. It can also be used to install flux to the cluster if the repository already exists.

Upgrading

When a new version of flux comes out, you can upgrade and it will carry over to the cluster. It’s really easy, it just upgrades its own files so when those are committed it will pull in the new version of the controllers.

Trouble

Aggregation yaml

For starters here is a minor one that took way too long to diagnose. I forget exactly what this is called but essentially it is a yaml file says run deal with the listed resources. Imagine the following directory contents:

drwxrwxr-x 4 angle angle 4,0K Αυγ  25 01:23 .
drwxrwxr-x 5 angle angle 4,0K Ιουλ 21 03:42 ..
drwxrwxr-x 2 angle angle 4,0K Αυγ  13 15:55 helmrepos
-rw-rw-r-- 1 angle angle  219 Ιουλ 21 03:20 infra-source.yml
-rw-rw-r-- 1 angle angle  126 Αυγ  12 11:45 kustomization.yaml
drwxrwxr-x 2 angle angle 4,0K Αυγ  13 15:56 storage

The kustomization.yaml file is as follows:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - infra-source.yml
  - helmrepos
  - storage

So it’s a way to more easily include stuff. The issue I had is that I named the file kustomization.yml instead of kustomization.yaml which for some reason isn’t supported. I believe this is a bug and not intended behavior but it was still frustrating to find and fix.

Logs

Now on to more substantial issues. While troubleshooting I got a bit frustrated with how flux displays errors. The messages are fairly non-descript and there isn’t a lot of guidance for how to fix them.

Conclusion

Despite a couple rough edges, flux has been working incredibly well. Even before I got it fully functioning, it was amazing to be able to just install it to a newly creted cluster and have all my applications running just like that. It’s almost magical and I’ve been enjoying it quite a lot. If you aren’t using a continuous deployment tool, feel free to give flux a try or read the previous post to learn about some other options

Fluxing My Cluster

Sat, 17 Apr 2021 17:24:22 +0300

It’s no secret that I’m a fan of automation and making life easier (even if I made it harder in the first place). One of the issues I’ve been having in my homelab is dealing with deploying stuff to kubernetes. Initially I wanted to just add a plugin to drone and be done with it. However, that didn’t really pan out which ended up being to my benefit since I discovered flux.

The problem

The cluster in its current state is a little sad because I end up doing too many things manually. Whenever something changes it’s time to bring out ye old kubectl apply -f in a local clone of the repo and that doesn’t spark joy. This was obviously bad form and couldn’t continue. What you also have to keep in mind is that I’ve chosen a 3rd-party storage plugin, democratic-csi and assume it’s the default storageclass in the cluster. This means that on a new cluster it’s required that before anything else, that one is added as a storageclass and set as default and local-path is removed from being a default. After that, the applications can be deployed which includes a mix of stuff I’ve written for the purposes of testing as well as 3rd-party software. Now that the stage has been set, let’s see what I tried

Solutions

Drone CI plugin

Initially the idea was that since I’ve alrady solved this for code using drone for CI (this will be discussed in a later post), I could just use a plugin and be done. However, my experience with writing helm charts is limited and the only official plugin for deploying to kubernetes is for helm charts, not kubernetes manifests. Furthermore, as I mentioned previously, not all software deployed on the cluster is something that I have written so even if there was an official kubernetes manifest plugin it wouldn’t cover 3rd-party software. At this point, I was thinking of finding some all-inclusive CI/CD software that would cover everything I wanted to do. It took no more than 10 seconds of thinking to realize that I was looking for non-modular software and discard that idea.

Standalone Continuous Delivery

This was the obvious choice but the software in this category is plentiful. The main competitors were flux, argo and tekton. Their designs differ quite a bit so there is going to be quite a bit of opinion in the following analysis so just keep that in mind. The few things they had in common were that they were all advertised as cloud-native with support for kubernetes, had a cli and were written in Go.

ArgoCD

I started with argo because one of the people that wrote a really good blog post on using kubernetes at home mentioned that they use it so I thought I’d try it first. The installation process required a massive yaml file that you just curl | kubectl apply -f which I wasn’t a big fan of but alright whatever. It also had a weird unlock procedure like jenkins where you have to find a token generated at runtime to unlock it but that’s okay, just an one-time setup thing. After spending a couple evenings with it, I wasn’t satisfied with the experience. The docs weren’t really good at explaining all the argo custom resources and getting a basic single-pod application running took me more than 2 hours. It was doable but this was going to be something I had to do for every application running on my cluster and the prospect didn’t seem appealing. I’m sure with more time I could maybe become more familiar and eventually warm up to it however first impressions were bad so I decided to move on.

Tekton

Tekton’s marketing was interesting so I took a look at it next. It seemed to be easier to set up and a couple random blog posts seemed to praise it so I downloaded another massive installation yaml which the guide told me to just kubectl apply -f which is okay I guess. I spent a few hours with it and it became clear you basically had to use it both for CI and CD so I put it to the side and moved on.

Flux

So why did I mention this one first but try it last? A few reasons. While researching, the v1 -> v2 development effort was going on and v2 didn’t have a lot of features but v1 was being phased out so I didn’t know if I should spend time learning v1 while v2 was coming along or if it’s worth jumping into alpha/beta stage software to avoid using legacy versions. By the time I did get around to it though v2 was clearly the way forward. Not only was it officially endorsed but among people that self-host, it was already the way to go 6-12 months ago. The docs were pretty clear, the cli was cross-platform and could be installed on a Raspberry Pi (something that argo at the time didn’t have available) and the examples worked. In an evening I had learned basically everything I needed to in order to just drop it in, generate the flux CRDs for my existing kubernetes manifests as well as helm charts and be up and running. The only issue I had, probably just a pet peeve of mine, was that the repo it creates on github is private by default and my goal is to have everything public and also secure (not there yet) but whatever. Flux v2 also mentioned as the gitops toolkit is made up of several different parts that have a specific purpose which is something I certainly appreciate. All in all, I think I’m going to stick with flux but you never know how homelab things will end up.

Conclusion

Flux is a really great project and I’ve enjoyed using it so far. The problem of continuous delivery has been solved for me and I recommend checking out if you’re in a similar position. Next up, I’m going to look at renovate bot for automating image updates, SOPS for secret management (which flux supports and an upgrade controller) to further help automate operations. Thank you for reading, I hope you enjoyed it and maybe learned something.

Continuous Deployment on inherent site

Flux Update

Long(ish)-term experience

Successes

Deploying raw manifest applications

Command-line tool

Upgrading

Trouble

Aggregation yaml

Logs

Conclusion

Fluxing My Cluster

The problem

Solutions

Drone CI plugin

Standalone Continuous Delivery

ArgoCD

Tekton

Flux

Conclusion