Kubernetes on inherent site

Remote Access Homelab

Sun, 04 May 2025 18:26:25 +0300

One of the remaining things for my homelab to have is remote access. I set up a couple options and faced some issues during the process which I want to document here.

Tailscale

Tailscale is wireguard-based VPN solution. Its main idea is that you run it alongside whatever you want to be accessible in your tailscale virtual network and then you can get to it. My primary concern was replacing something like openvpn where it can be set up so it’s almost like being connected to the local network. In tailscale this can be achieved by making it announce that it can route to a specific subnet. This way things like local DNS works normally (for example I can access opnsense.home.inherently.xyzand see the web interface of my router). Additionally, you can make your traffic be routed through a specific device. This might be the router at home which will be marked as an exit node and then in the mobile app select it to be used as such. The choice of tailscale dates back to when I was still running pfSense. In pfSense there has been a built-in integration for quite a while that can work as an exit node and subnet router. In OPNSense it used to require a very manual install and setup process through the FreeBSD ports system but now there is a plugin which is a lot more user-friendly. I set it up that way and then joined my phone and my tablet to the tailnet which means I can access all of my servers remotely. Problem solved, let’s go home, roll the outro. Not yet. The thing is that while this works fine for just myself, if I want someone to check out some new selfhosted service I have or connect to a database I’m hosting I don’t want them to have to set up tailscale first.

Cloudflare Tunnel

To address that need, I chose Cloudflare Tunnel. This works a bit differently than Tailscale, you run cloudflared and tell it where traffic for a domain name should be redirected. In my case it runs inside kubernetes and points to the ingress controller. This… took a while for me to get working. It was mostly my own fault and lack of knowledge of how cloudflare works. There are certificates that cloudflare generates which encrypt traffic between clients and cloudflare. This process is done automatically and only for the root domain, for any subdomains you better get your wallet out. This meant that since I was trying to access services that were something like service.home.inherently.xyz, TLS was broken. Took about 2 days to figure this out and I only managed it thanks to Kryptonian in the Home Operationsdiscord server. The way I fixed the TLS problem was by just putting everything under the root domain which then made it work. There is a bit more work to do because I think cluster-template does have a good idea with separating services that should be externally accessible by running two ingress controllers. However, at this point it gets the job done so I’m reasonably happy with it.

Conclusion

Access do be external.

Talos Linux Setup and Configuration

Tue, 11 Mar 2025 19:28:45 +0200

After about two weeks since the first time I installed Talos, I think I’ve figured out a good setup. In the previous post I went over the PXE setup more than the kubernetes part of it so now I’ll cover the other half.

How the Talos configuration works

Following the getting started guide tells us that to generate the configuration files we need to run something like talosctl gen config homecluster https://10.0.50.69:6443. This command will generate 3 files. These are controlplane.yaml, worker.yaml and talosconfig. The control-plane nodes will use the first file, the worker nodes will use the second file and the third one should be placed into ~/.talos/config (with an edit to specify the node IP addresses or as Talos calls them in this case, endpoints). If we run the command we’ll get some output like this:

generating PKI and tokens
created controlplane.yaml
created worker.yaml
created talosconfig

Then, if we try to re-run the same command we will get a message like the following:

1
2

generating PKI and tokens
file "/home/angle/talos/controlplane.yaml" already exists, use --force to overwrite

If we do the “just go away and do it anyway” thing that it says and add --force it will again show the previous output. In this process, we’ve wiped all the secret tokens and certificates it generated before and cannot recover them. For a first-time installation this is barely an inconvenience, we probably do not care. When doing this for an existing cluster, we will also need to get a new kubeconfig since the secrets that are re-generated are also relevant for the certificate that is stored in the kubeconfig to access the cluster. That’s one of the reasons that if you’ve had the cluster running for a couple months, this might be a bit more annoying.

Separate secrets

The alternative is to take a more in-depth look at the documentation which will land us on a page with notes for production-level setups, specifically the section about separating out secrets. Following the examples in that page we can run something like talosctl gen secrets -o secrets.yaml first and then talosctl gen config --with-secrets secrets.yaml homecluster https://10.0.50.69:6443. This way, whenever we run the command to generate the configuration files (controlplane.yaml, worker.yaml, talosconfig) we will not also wipe the secrets. Now, why is that process useful or even preferable?

Using this approach, the secrets can be stored separately from the configuration itself, that’s the obvious part. The less obvious part is that given the same talosctl, the CLI tool, we can generate the same 3 files. Additionally, the kubeconfig that we can receive by running talosctl kubeconfig ~/.config/kube/config --nodes 10.0.50.72 --endpoints 10.0.50.72 is not going to need to be updated even if we reinstall Talos. You might then think, if we continually re-generate these 3 files, how can we ever permanently store configuration changes? Putting them directly in controlplane.yaml and worker.yaml is a no-go since they’ll be overwritten by a talosctl gen config command. That’s where patches come in.

Patches

The talosctl CLI tool has a couple helpful options for applying patches when generating configuration. Similar to having one configuration file for control-plane nodes and one for workers, the CLI flags are the same way. There are 3 flags: --config-patch-control-plane which applies the patch only to control-plane nodes, --config-patch-worker which applies the patch only to worker nodes and --config-patch which applies it to both. Using these, we can write patches and apply them as we wish to any and all of the nodes in our cluster. All 3 of the flags accept either inline JSON patches or YAML files depending on the notation. Now I’ll switch gears a bit and discuss my own specific setup as an example. I have 3 patches at the moment. First, is a baseline that configures network interfaces, defines the installation disk, sets up NTP, adds kernel parameters, enables workload scheduling on the control-plane nodes and certificate rotation for the kubelet. The other two are specific for the openebs storage system and for the metallb bare-metal loadbalancer. It’s important to note at this point that it merges the changes. So if you want to remove something from the default configuration (such as the node.kubernetes.io/exclude-from-external-load-balancers: "" node label) it’s not as easy as just having the key specified but being empty (for example nodeLabels: {}). To generate my configuration, I can run the following:

talosctl gen config homecluster https://10.0.50.69:6443 \
	--with-examples=false \
	--with-docs=false \
	--with-secrets=secrets.yaml \
	--config-patch=@patches/01_baseline-changes-patch.yaml \
	--config-patch=@patches/02_openebs-mayastor-patch.yaml \
	--config-patch=@patches/03_metallb-patch.yaml \
	--endpoints=10.0.50.71,10.0.50.72,10.0.50.73

Here I’ve disabled documentation and examples just so the file isn’t as cluttered. The important part though is that the sensitive information is only stored in only one file which could be encrypted and put into version control. From there, I could use the same version of the CLI tool to generate the final configuration file. The goal is to be able to have some sort of GitOPS setup where all or most of what’s needed to create the Talos configuration is stored in the same infra repository as everything else. One tool that seems to fit this exact use-case is talhelper but I’ve not looked into it yet.

Conclusion

I keep finding cool stuff about Talos and the homelab tinkering will continue until morale improves.

PXE setup for Talos Linux with Matchbox

Sat, 01 Mar 2025 19:00:33 +0200

Join my campaign of hate against installing operating systems interactively. I usually wipe my homelab virtualization hosts every week or two. This might sound insane but I’ve been working on automating their setup. As explained in the previous post, friendship ended with ProxMox and kubernetes will take its place. Specifically Talos Linux, a linux distro you can’t even SSH into. ProxMox kind of notoriously doesn’t have a way to be sanely installed through PXE but Talos is kind of made for it. However, PXE (and iPXE, we’ll discuss this later) is not perfectly documented and can be even harder to debug. So in this post I’ll document what worked for me and talk about some pretty helpful resources.

Plan

PXE

So let’s start with a bit of background, then lay out what I wanted to achieve, why and what I considered “good enough”. First off, PXE. It stands for pre-boot execution environment and is mostly reliant on motherboards and network cards to be implemented on the client side. It’s a stripped down environment before the computer is booted into an operating system. Inside it, there are standards for getting information about what to execute, where to get it and what configuration to use. PXE is the name of the general concept and the first version of the technology. It allows booting a computer from the network by loading files from a TFTP server. Later on, iPXE came around to fix some of its limitations. One thing it’s useful for having a computer and being able to boot it from the network and install an operating system on it without having to have a USB stick on you. With something like syslinux you can even create menus to be able to choose what to install and have a bunch of different options with different settings. As computer installations grow larger, it’s easy to see how this becomes kind of essential as you can’t run around multiple datacenters any time something needs to be re-imaged. A counter-point to that could be that servers have remote management features allowing us to load ISOs as if they’re DVDs but even then, iPXE has an advantage because you can fully automate the process. Also my poor version of servers (read: used office PCs) do not have IPMI. At any rate, that’s the general idea of PXE.

Talos

How does that apply to my case then? I want to be able to completely wipe all three systems and then effortlessly restore them. As I plan to run kubernetes on it, the installation of applications to the cluster is taken care of using FluxCD so I only really need to solve the installation part. That’s where Talos comes in. Talos is an incredibly minimal linux distribution that is designed with the sole purpose of running kubernetes. No SSH, no TTY, no nothing. Sitting in front of the system only allows you to change kernel boot parameters and the network configuration. Everyhing is done using the API which is accessible interactively through the talosctl CLI utility. This means that we can have our operating system and kubernetes configuration as code for the price of one config file. Sign me up!

Matchbox

Those are the ingredients but it can hardly be called a meal. We need to get some kitchen utensils in this analogy to get the desired result. Our PXE setup will need to have a little bit of logic in it so it knows what to install with what configuration. See, Talos can be told to fetch a configuration file from an HTTP server using a kernel parameter. A conditionally-rendered file would allow us to alter the response depending on who’s requesting it. This is where matchbox slots in. When matchbox gets a request it can see some information about the host requesting it and therefore make a determination about what to reply with. We can then serve different options according to which system is asking such as what Talos configuration file to use. This way the control-plane kubernetes nodes can receive the Talos controlplane.yaml while worker nodes get the worker.yaml. So a brand-new host with an empty disk drives would boot up, skip the disks since they don’t have an operating system and then try booting using PXE. Then the host would get its configuration, install the operating system, install kubernetes and become part of a cluster.

Definition of Done

The ideal process when getting adding a host should look like:

note down the MAC address and corresponding IP somewhere
plug in the cables
configure the boot device order
save settings and exit
play minecraft while waiting
run one or zero commands
host shows up in kubectl get nodes
recreating it from scratch should be just as easy

I don’t know if I can get exactly to that point but I’m willing to put in some effort into the setup so in the future I can avoid it to a greater extent. Some compromise is acceptable since I don’t have any IPAM infrastructure in place already so this isn’t the full and proper way to go about it.

Implementation

How do I do X in Y minutes

Meat and potatoes first, for quick reference here is what I did (5-6 hours of suffering omitted for brevity):

Installed tftp-hpa on Arch
Downloaded undionly.kpxe and ipxe.efi
Put them in /srv/tftp

Created matchbox.ipxe with the following contents:

1
2

#!ipxe
chain http://10.0.20.50:8080/boot.ipxe

Installed matchbox using Docker by running the following command:

`1`	`docker run --net=host --rm -v /var/lib/matchbox:/var/lib/matchbox:Z -v /etc/matchbox:/etc/matchbox:Z,ro quay.io/poseidon/matchbox:v0.10.0 -address=0.0.0.0:8080 -log-level=debug`

Downloaded vmlinuz-amd64 and initramfs-amd64.xz
Put them in /var/lib/matchbox/assets
Download talosctl
Run talosctl gen config homecluster https://10.0.50.69
Edit controlplane.yaml until it looks good
Put controlplane.yaml in /var/lib/matchbox/assets as well

Create /var/lib/matchbox/profiles/control-plane.json:

{
  "id": "control-plane",
  "name": "control-plane",
  "boot": {
    "kernel": "/assets/vmlinuz",
    "initrd": ["/assets/initramfs.xz"],
    "args": [
      "initrd=initramfs.xz",
      "init_on_alloc=1",
      "slab_nomerge",
      "pti=on",
      "console=tty0",
      "printk.devkmsg=on",
      "talos.platform=metal",
      "talos.config=http://10.0.20.50:8080/assets/controlplane.yaml"
    ]
  }
}

Create /var/lib/matchbox/groups/control-plane.json:

{
  "id": "kube01",
  "name": "kube01",
  "profile": "control-plane",
  "selector": {
    "mac": "18:03:73:2a:d8:a2"
  }
}

In OPNSense (link to my router)
- Under DHCP Static Mappings for this interface. add the hosts
- Under TFTP server:
  - Set TFTP hostname: 10.0.20.50
  - Set Bootfile: undionly.kpxe
- Under Network booting
  - Set next-server IP: 10.0.20.50
  - Set default bios filename: undionly.kpxe
  - Set iPXE boot filename: matchbox.ipxe
Boot Optiplex
Wait
Watch text scroll on screen
???
No profit, I spent my whole weekend on this

Explanation

PXE is an under-documented part of the common network services one can set up. An amusing yet educational thing I found was a redditor’s 3-part descent into madness and consequently enlightenment:

The matchbox documentation was also very good at explaining both the general process and giving some specific recommendations:

With those out of the way I’ll make an attemt at explaining it myself. When a PXE client starts, it does the DHCP dance which tells it where the TFTP server is and what the name of the file it should execute is. That file is a network boot program which can optionally load some configuration (also stored on the TFTP server). So after all of that is figured out, the PXE client will pull the kernel and initramfs, again from the TFTP server. In contrast, iPXE uses scripts stored on the TFTP server instead of plain configuration files so we can do fancier things with it. However, my old desktop PCs do not have iPXE so I had to use an in-between thingy, undionly.kpxe. What that does is load iPXE firmware (ipxe.efi) so we can access the cool new features without having to replace any hardware. When that loads, it talks to the DHCP server as an iPXE client this time and gets told to grab the matchbox.ipxe file from the TFTP server. That file is an iPXE script which tells the client to check the matchbox server. When matchbox gets the request, it checks the groups it has registered and if there are any matches according to the selector field, it uses the corresponding profile. In this case, the profile is for control-plane Talos Linux nodes. The kernel and initramfs listed in the profile are used alongside some specific kernel parameters. The most important parameter is talos.config=http://10.0.20.50:8080/assets/controlplane.yaml which tells the machine that is booting the operating system to fetch the configuration file listed there. Of course, you will need to have generated the configuration (using talosctl gen config homecluster https://10.0.50.69:6443) and moved it in the matchbox assets directory (or in another HTTP server, matchbox is just convenient in this setup). That configuration has some basic things like which disk and network settings to use. When the machine gets this configuration, it installs Talos and then waits. At this point the system is booted normally, no PXE involved.

Following that, all we need to do is run talosctl bootstrap --nodes 10.0.50.71 --endpoints 10.0.50.71 and we get a kubernetes cluster. To get the kubeconfig required to access the cluster, run talosctl kubeconfig my-talos-kubeconfig --nodes 10.0.50.71 --endpoints 10.0.50.71 and then check that it works by running kubectl --kubeconfig my-talos-kubeconfig get nodes. I’ve tested the whole process end-to-end twice now to ensure that nodes with blank disks and correct boot order will behave correctly and it’s almost magic. True enough, after pressing the power button on all three systems, making some coffee, and then running the commands mentioned previously, I got this output:

$ kubectl get nodes -o wide
NAME     STATUS   ROLES           AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION   CONTAINER-RUNTIME
kube01   Ready    control-plane   3h20m   v1.32.2   10.0.50.71    <none>        Talos (v1.9.4)   6.12.13-talos    containerd://2.0.2
kube02   Ready    control-plane   3h20m   v1.32.2   10.0.50.72    <none>        Talos (v1.9.4)   6.12.13-talos    containerd://2.0.2
kube03   Ready    control-plane   3h20m   v1.32.2   10.0.50.73    <none>        Talos (v1.9.4)   6.12.13-talos    containerd://2.0.2

And I guess that’s it. It needed some effort and research and the approach should be adjusted according to the network situation (the matchbox docs saved the day here). In the end though, I really like what I got working and it was a fun way to spend the weekend. I can go from basically brand new computer with nothing on it to full cluster with a high availability control-plane using patience and a single command. Two commands and I can install fluxcd on the cluster which will install all my applications. Looking back at the definition of done I set in the beginning, I consider this a success.

Conclusion

As the proud owner of a Talos Linux kubernetes cluster installed in a fully automated way, I am quite happy. I can get back into running services at home and live through the pain of using kubernetes daily again. I have a couple of ideas in mind for how to use my new container running platform but that’s for another post. Hopefully you got a better idea of what PXE is, how it works and how I used it in this instance. If you weren’t familiar with any of this, maybe it’s time to get your hands dirty and prepare to throw away your USB drives. Either way, that’s all for now.

Homelab 2025 Plans

Fri, 28 Feb 2025 22:51:33 +0200

Intro

After recapping the state of my homelab and writing down how it was in the end of 2024, I think it’s time to look towards the future. Having finally finished my thesis, I now have less constant stress to deal with and a bit more time than I’ve had in the last 2 and a half years. The main goal of rebuilding my homelab is to make it an actual production environment. This means that a few important problems need to be solved before it’s considered done. Along with that, I’m willing to lose a bit of flexibility in exchange for having something that actually works which is a bit odd for me to write but alas.

Hardware

Over the past several years I’ve invested quite a bit into the hardware that already exists. I won’t go into great depth about it, all the details are in the previous post. My homelab is housed in a 27U rack, includes a managed switch, a DIY router, 3 virtualization hosts and at least one NAS. That is more than enough to at least get started running several services. I will try to avoid investing in hardware this year with two exceptions, a VGA KVM and a good NIC for the router. Other than those two additions, I plan to focus on software choice and configuration.

Software

Kubernetes

I’m going bare metal kubernetes. This might seem counter-intuitive but it’s actually the easiest thing to get working. I still like ProxMox, I’m happy with the terraform provider for working with ProxMox and there is now a healthy alternative one, I use ansible at work and it’s all nice but I’m looking for an even higher level of automation. The integrations available for kubernetes are at a great point and have been for a couple of years now. I can forego dealing with issues such as “how do I make DNS automatically update when I start running a new service” and focus more on running and using my self-hosted services. Of course, kubernetes presents its own set of challenges and has its own quirks but I believe the tradeoff to be worth it for me.

The plan right now is to go with Talos Linux so my hosts are nice and lean. The entire process of installation can be automated using the matchbox PXE server and a bit of configuration and the operating system has just enough to run kubernetes and that’s it. I could of course regret this in which case I’ll go back to Debian my beloved but it remains to be seen. Initially I’ll go with a simple ISO installation to get started and then I’ll see if I want to take it a step further. The kubernetes-related stuff is mostly figured out, default CNI, nginx for ingress, metallb for external IPs, flux to deploy stuff and I’ll try adding renovate bot to help with maintenance. What remains is to decide if I’m going to go with openebs or longhorn for storage but that shouldn’t be too difficult or time-consuming.

Router

The other interesting software choice is the router. Having a dedicated machine for it seems wasteful and I’m bound to running whatever software is available for FreeBSD. No, I’m not planning on moving away from OPNSense at the moment but I do want to start the vicious cycle again. The plan is to have it in a VM, thus the vicious cycle, and then be able to run things like matchbox on the same machine, a different NTP server, a different DHCP or PXE server (mainly tinkerbell comes to mind), different DNS server (Hickory seems quite interesting) and so on. Those could be containers inside other VMs or whatever else, I’m not bound to whatever is packaged for FreeBSD or even specifically for OPNSense. This of course requires a hypervisor (ProxMox makes a comeback) and a NIC that can be passed through to a virtual machine in order to run the router. This is a secondary concern since I do have an old machine from the core2duo days if I just want to runs some basic network infrastructure like the ones mentioned previously for experimentation purposes. VyOS has also been on my mind but that’s a bigger jump and would require quite a bit of effort to adapt everything I have to it. The router plans are unclear at the moment so it’s more of a backburner project.

Conclusion

One must imagine Sisyphus, he rolls the rock and I rebuild my homelab. Was there a point to writing this? I’m not sure. These are my plans at the moment so I can get back into the hobby. If I do go through with them, I’ll try to document the experience when I have something workable.

State of Homelab 2024

Mon, 27 Jan 2025 22:38:48 +0200

When operating a homelab there is always something new to do or something waiting to be fixed. This post’s aim is to detail the state of my homelab at the end of 2024 and beginning of 2025.

Router

The previous post was about virtualizing pfSense inside ProxMox but this setup is no longer in use. In its place, there is a new system running OPNSense on bare-metal. This was done for a few reasons, the main one being that I wanted to move away from pfSense. More than that, there was a circular dependency where the ProxMox host had to be up in order for the router to function but in order for the cluster nodes to talk to each other, the router needed to be online. In practice this was not a huge issue but it bothered me since it made experimenting more difficult because recreating the cluster meant taking down the router as well. I’ve lost a few things, mainly pfblocker-ng and native tailscale integration but they’re not as big of a loss as I had anticipated. In the end I think it was worth it, over the course of the past few years the company making pfSense has acted against the best interest of pfSense users and I did not feel like using their software anymore.

Virtualization

I’m still running ProxMox, although now it is a 3-node cluster. I’ve spent a pretty significant amount of time tweaking the bootstrap process from a fresh install to having the entire homelab operational. Somewhere along the line I tried making it so non-cluster operation of multiple hosts was supported but gave up on that eventually. Each of the nodes has a 250GB SSD used as a boot drive and for node-local storage along with a 2TB HDD used as a Ceph OSD. Since the last post about using ansible and terraform with ProxMox, I’ve figured out the process of adding tools to the debian image in the virtual machine template. There is also the option to deploy using GitLab CI so just making committing and pushing to the repo is enough to create a new virtual machine. The k3s virtual machine terraform configuration was improved significantly but I’ve not adopted modules yet which seems to be the next step for organization-related improvements. I also run all software directly on top of debian virtual machines without containers to practice writing ansible. At work, our team runs everything in Docker so it could be a good idea to start moving to something similar at home for the sake of practice.

Kubernetes

The kubernetes cluster got a significant update. I’ve adopted FluxCD to a greater degree, figured out secrets storage and a pretty good structure for the YAML files. The k3s cluster has 3 control plane nodes and 3 worker nodes, one per virtualization host. I’m using longhorn for storage with plans to try and use the Ceph cluster as an additional storage class. The networking setup consists of MetalLB to have an IP in the subnet and then run ingress-nginx as the ingress controller on that IP. No plans for checking out Gateway API have been made at current but it is a good and necessary step forward in the ecosystem, I plan to move to it after things are a bit less in flux. The DNS is manual right now due to the low amount of services but I’ve tested external-dns with PiHole and I’m aware of the OPNSense webhook option, it remains to be seen what I’ll end up using. The setup needs a bit more tender love and care but it’s running well and doing what I need it to. I have plans to move more stuff to the cluster after the foundation is solid and not just have my own demo applications in there. The thought has crossed my mind to completely do away with ProxMox and then run Talos Linux bare metal on the systems but I’d be losing out on some flexibility which I don’t want to commit to right now. All in all, kubernetes is still pretty fun and the 1.30 and later releases have made good quality of life improvements.

NAS

Since I upgraded my desktop, the old one’s internals were transplanted to a different case are now a second NAS. I’m saving up to get a big external drive so I can have an extra copy of the current main NAS so I can freely mess with them to have a solid backup solution. The changes in TrueNAS Scale regarding applications has left me waiting for things to settle down before I commit to a final setup. Overall nothing exciting on this front.

Hardware

I run mostly used office desktops and a basic managed switch. I have 3 Dell Optiplex machines: 2nd gen, 3rd gen and 4th gen Intel i5 CPUs with RAM sizes ranging from 16 to 32GB each. For storage they have a 250GB SSD and a 2TB HDD as mentioned earlier. The router is a Fujitsu equivalent desktop with a 4th gen Intel Pentium and 8GB of RAM as well as a 500GB SSD. My rackmount NAS is an i5 4570 with 16GB of RAM and 4x4TB HDDs in a 3-drive ZFS RAID-Z1 plus one hot spare. My backup NAS is an i7 4770 with 32GB of RAM and 2x8TB HDDs in a 2-way ZFS mirror. The main network switch is a Netgear GS724TV4, just a layer 2 managed gigabit switch.

Conclusion

I wanted to write all of this down to have a reference for myself when I inevitably change something and it might help someone else as well. Also the blog is back(?) after more than a 2 year silence, hopefully I will not neglect it for that long again.

Flux Update

Wed, 25 Aug 2021 00:44:34 +0300

I wrote about my journey of choosing a continuous deployment tool and why I ultimately ended up using flux for my homelab in this post. It took quite a bit of reasearch and if you’re in a similar position as I was it might prove to be helpful.

Long(ish)-term experience

I’ve been using it for about 5 months now, the first post was written about one month of use. At that point I had been tinkering with it and attempting to get all the stuff I wanted to be deployed by it. Some things were simple, others not so straight-forward.

Successes

Deploying raw manifest applications

Deploying raw kubernetes manifests was really easy. Since I’m somewhat on the fence about Helm, I prefer using normal kubernetes manifests where possible. This meant that for my use case mostly everything was smooth sailing after figuring out how to generate a kustomization yaml using the command-line tool.

Command-line tool

Speaking of that, flux has a command-line tool to interact with the controllers running inside the cluster. It’s also the way you can initialize a repository to be used as the place for flux files to be stored. It can also be used to install flux to the cluster if the repository already exists.

Upgrading

When a new version of flux comes out, you can upgrade and it will carry over to the cluster. It’s really easy, it just upgrades its own files so when those are committed it will pull in the new version of the controllers.

Trouble

Aggregation yaml

For starters here is a minor one that took way too long to diagnose. I forget exactly what this is called but essentially it is a yaml file says run deal with the listed resources. Imagine the following directory contents:

drwxrwxr-x 4 angle angle 4,0K Αυγ  25 01:23 .
drwxrwxr-x 5 angle angle 4,0K Ιουλ 21 03:42 ..
drwxrwxr-x 2 angle angle 4,0K Αυγ  13 15:55 helmrepos
-rw-rw-r-- 1 angle angle  219 Ιουλ 21 03:20 infra-source.yml
-rw-rw-r-- 1 angle angle  126 Αυγ  12 11:45 kustomization.yaml
drwxrwxr-x 2 angle angle 4,0K Αυγ  13 15:56 storage

The kustomization.yaml file is as follows:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - infra-source.yml
  - helmrepos
  - storage

So it’s a way to more easily include stuff. The issue I had is that I named the file kustomization.yml instead of kustomization.yaml which for some reason isn’t supported. I believe this is a bug and not intended behavior but it was still frustrating to find and fix.

Logs

Now on to more substantial issues. While troubleshooting I got a bit frustrated with how flux displays errors. The messages are fairly non-descript and there isn’t a lot of guidance for how to fix them.

Conclusion

Despite a couple rough edges, flux has been working incredibly well. Even before I got it fully functioning, it was amazing to be able to just install it to a newly creted cluster and have all my applications running just like that. It’s almost magical and I’ve been enjoying it quite a lot. If you aren’t using a continuous deployment tool, feel free to give flux a try or read the previous post to learn about some other options

Droning On

Sat, 01 May 2021 18:54:59 +0300

I described previously how the deployment of applications is automated in my homelab but didn’t touch upon how I test my own code. The topic came up when I started trying to simulate a development pipeline who’s first step after committing code is to be run through the continuous integration system. What should that system be though? In this post, I’ll take you through the things I tried first and then why I went with drone in the end.

Choose a Fighter

My quest is to ultimately end up with the skills to set up kubernetes so that it can be deployed to everything from a DIY NAS and a stack of NUCs to a warehouse full of servers backed by distributed storage. I’m obviously more towards the first camp so that’s where my journey is at. With that said, my goal is to have software small and lightweight enough that it can run on the former setup easily so that there can be multiple instances of it in the latter form. The current mental model I have is developer organization with not much more than 3 NUCs + 1 NAS per team without losing the ability for continuous integration and continuous deployment. Therefore the landscape needs to be explored and the obvious titan of the old world is jenkins with many newcomers eager to take its place. The obvious caveats still apply, it must be open-source, able to be self-hosted, support for gitea is needed and being able to run well on kubernetes is a foregone conclusion.

Jenkins

It should need no introduction, most developers are familiar with the trusty crusty old butler software. Jenkins is probably one of the most prominent pieces of CI/CD software that was pushed to the forefront when the push for more automation and increased code quality really happened. Sadly there are a few reasons that annoyed me from the get-go. First, the unlocking procedure requires you to kubectl logs <podname> in order to find the unlock key which I understand but nonetheless find annoying. Second, the Jenkinsfile has its own domain-specific language (DSL for short) that I find unappealing even though you won’t catch me praising YAML’s lack of curly braces. I begrudgingly went though it and wrote a Jenkinsfile to give it a fair chance but it wasn’t a nice experience. Next, a single instance uses too much ram to the point that it can barely fit in my k3s VMs which use 2.5GB of ram each together with other software. This is an issue because I like running at least 2 instances of most applications to be sure that everything works correctly when scaling up. Last but not least, it’s a 2-in-1 solution which I don’t find appealing as far as architecture and since I was planning to find a standalone piece of software to do CI and another one to handle CD, it was a hard sell from the start. As a redeeming quality it has an official kubernetes pipeline plugin which is nice so they’ve at least put thought into this use case. Ultimately it wasn’t the right fit for me so it was time to move on.

Drone

While pondering this I remembered an old post from one of the blogs I read, specifically this one. I said what the hell, why not and started looking its website. Drone is a very nice and simple continuous integration tool. From the start it seemed like it would be a good pick but you never know until you try. I wrote some kubernetes manifests, got angry until I got them to work and a half hour later it was hooked up to my local gitea instance and ready to use. I added a .drone.yml to one of my repos, pushed the change, activated the repo from the drone dashboard, made another change to the source code, pushed it and waited to see what happened. Instantly drone received the webhook event and got to work. The dashboard is pretty simple (and sadly lacks a dark mode) so you just hit the thing you want to look at, click on the build and see it happen. It supports different things running at the same time like databases if you want your tests to include that which can be run either as what drone calls services aka run in the background from the beginning or if you want an ordered dependency you can use detached steps which is especially useful in microservices. My only real complaint aside from the lack of dark mode is the inability to trigger a build manually without pushing but that’s only an issue when you first activate a repository inside drone’s dashboard. All in all, it has been a joy to use, most of the projects I am currently developing have a .drone.yml and get built at least 3 times a day when gitea mirrors the gitlab or github repo they’re mainly hosted at.

Conclusion

This wasn’t as big of a journey as finding a continuous development tool because really I only found the two aforementioned options that had the basics I was looking for and one of them didn’t really fit the bill. I’m not huge into software development, it’s mostly a hobby for me as might be obvious by now, since my main concern is infrastructure-related stuff. With that said it’s useful to have the skills to build an application that can communicate with the kubernets api to automate something or write an ansible module and most of all it allows me to see the developer perspective. I haven’t started looking into SOPS or renovate bot or the k3s upgrade controller or velero but I hope to get the chance to do so in the future. Thank you for reading, I hope you enjoyed it and maybe learned something.

Fluxing My Cluster

Sat, 17 Apr 2021 17:24:22 +0300

It’s no secret that I’m a fan of automation and making life easier (even if I made it harder in the first place). One of the issues I’ve been having in my homelab is dealing with deploying stuff to kubernetes. Initially I wanted to just add a plugin to drone and be done with it. However, that didn’t really pan out which ended up being to my benefit since I discovered flux.

The problem

The cluster in its current state is a little sad because I end up doing too many things manually. Whenever something changes it’s time to bring out ye old kubectl apply -f in a local clone of the repo and that doesn’t spark joy. This was obviously bad form and couldn’t continue. What you also have to keep in mind is that I’ve chosen a 3rd-party storage plugin, democratic-csi and assume it’s the default storageclass in the cluster. This means that on a new cluster it’s required that before anything else, that one is added as a storageclass and set as default and local-path is removed from being a default. After that, the applications can be deployed which includes a mix of stuff I’ve written for the purposes of testing as well as 3rd-party software. Now that the stage has been set, let’s see what I tried

Solutions

Drone CI plugin

Initially the idea was that since I’ve alrady solved this for code using drone for CI (this will be discussed in a later post), I could just use a plugin and be done. However, my experience with writing helm charts is limited and the only official plugin for deploying to kubernetes is for helm charts, not kubernetes manifests. Furthermore, as I mentioned previously, not all software deployed on the cluster is something that I have written so even if there was an official kubernetes manifest plugin it wouldn’t cover 3rd-party software. At this point, I was thinking of finding some all-inclusive CI/CD software that would cover everything I wanted to do. It took no more than 10 seconds of thinking to realize that I was looking for non-modular software and discard that idea.

Standalone Continuous Delivery

This was the obvious choice but the software in this category is plentiful. The main competitors were flux, argo and tekton. Their designs differ quite a bit so there is going to be quite a bit of opinion in the following analysis so just keep that in mind. The few things they had in common were that they were all advertised as cloud-native with support for kubernetes, had a cli and were written in Go.

ArgoCD

I started with argo because one of the people that wrote a really good blog post on using kubernetes at home mentioned that they use it so I thought I’d try it first. The installation process required a massive yaml file that you just curl | kubectl apply -f which I wasn’t a big fan of but alright whatever. It also had a weird unlock procedure like jenkins where you have to find a token generated at runtime to unlock it but that’s okay, just an one-time setup thing. After spending a couple evenings with it, I wasn’t satisfied with the experience. The docs weren’t really good at explaining all the argo custom resources and getting a basic single-pod application running took me more than 2 hours. It was doable but this was going to be something I had to do for every application running on my cluster and the prospect didn’t seem appealing. I’m sure with more time I could maybe become more familiar and eventually warm up to it however first impressions were bad so I decided to move on.

Tekton

Tekton’s marketing was interesting so I took a look at it next. It seemed to be easier to set up and a couple random blog posts seemed to praise it so I downloaded another massive installation yaml which the guide told me to just kubectl apply -f which is okay I guess. I spent a few hours with it and it became clear you basically had to use it both for CI and CD so I put it to the side and moved on.

Flux

So why did I mention this one first but try it last? A few reasons. While researching, the v1 -> v2 development effort was going on and v2 didn’t have a lot of features but v1 was being phased out so I didn’t know if I should spend time learning v1 while v2 was coming along or if it’s worth jumping into alpha/beta stage software to avoid using legacy versions. By the time I did get around to it though v2 was clearly the way forward. Not only was it officially endorsed but among people that self-host, it was already the way to go 6-12 months ago. The docs were pretty clear, the cli was cross-platform and could be installed on a Raspberry Pi (something that argo at the time didn’t have available) and the examples worked. In an evening I had learned basically everything I needed to in order to just drop it in, generate the flux CRDs for my existing kubernetes manifests as well as helm charts and be up and running. The only issue I had, probably just a pet peeve of mine, was that the repo it creates on github is private by default and my goal is to have everything public and also secure (not there yet) but whatever. Flux v2 also mentioned as the gitops toolkit is made up of several different parts that have a specific purpose which is something I certainly appreciate. All in all, I think I’m going to stick with flux but you never know how homelab things will end up.

Conclusion

Flux is a really great project and I’ve enjoyed using it so far. The problem of continuous delivery has been solved for me and I recommend checking out if you’re in a similar position. Next up, I’m going to look at renovate bot for automating image updates, SOPS for secret management (which flux supports and an upgrade controller) to further help automate operations. Thank you for reading, I hope you enjoyed it and maybe learned something.

Homelab Current Form

Sun, 07 Feb 2021 14:35:32 +0200

The first part of the software changes I made was covered in the first part where I explained how and why I started going down this path.

Software choices

This whole journey was about making my home infrastructure better. Part of that was about having a way to more easily create the foundation on which kubernetes would run on, as well as describing my setup as code. Proxmox is a beast in its own right. Features include being able to manage kvm virtual machines and lxc containers, an API that can be used to interact with it programmatically and most importantly being open source. While researching what automation tools could interact with proxmox, I found a community-made terraform provider as well as a set of two ansible community modules (kvm and lxc). After thinking about it a little bit, I wanted to use terraform in a non-cloud environment but also gather some knowledge about it so that’s what I decided to use. I can have my provisioning requirements in a file that terraform understands and store it in git which is exactly what I was going for. Ansible is still in the picture since it is used to set up and configure the debian environment inside the virtual machines. Now that the hypervisor part is covered, let’s move on to running services.

Making it store is harder than making it run

As mentioned in the first part about software changes, I was learning kubernetes and now I could not only wipe and recreate the cluster itself but also the entire virtual machines that it was running on. Some time after starting to use proxmox with terraform I attempted to use rancher to manage kubernetes but ended up ditching it due to various problems with running even basic stuff on it (very likely that it’s a case of PEBCAK, I don’t think rancher is terrible or anything like that).

The problem

However the problem of shared storage continued to taunt me for the following months. There was seemingly nothing that a simple fella could do to use a nice and simple NAS running truenas core as storage that is able to be dynamically provisioned for use by kubernetes. Now, I hear you, “what about nfs-client-provisioner”, someone less familiar with this pile of madness might exclaim. Indeed it does exist and barely work except the helm chart for it is deprecated and it does not work with kubernetes version 1.20 and later since it does not seem to be using CSI drivers.

I really tried

Months of furious and frustrating testing ensued. Not only was I trying to get applications running on kubernetes, I was also fighting with the unexpectedly complex tast of using the storage server I had available.

glusterfs

I’ve gone through basically any and all commonly suggested options for dynamically provisioned storage. Prette early on I tried glusterfs with heketi by making 3 LXC containers and mounting an nfs share to each one that would serve as the brick in the gluster volume. Suffice it to say that it didn’t work and things were getting out of hand.

longhorn

After a bit more research and I found out that longhorn, another project by the authors of k3s, used iscsi to communicate with kubernetes. That could work I thought, except there was no coherent example of how to use it without using targetd for iscsi. Longhorn was starting to look more appealing, I could just put the virtual drives of the VMs on an nfs share, run longhorn inside the VMs to pool all their storage together and call it good enough. No, I could not give up yet. A sub-optimal solution would do if there was no other way but I was convinced something more was out there. My patience and hope were running out but not empty yet.

success

That all changed in early January where during my nearly daily search for possible storage solutions I hadn’t tried, I found out about democratic-csi. This was it. Made to be used with freenas, truenas as well as DIY ZFS setups. The silver bullet was here. Just a few minutes of reading about it and writing my configs, a short(read: long) helm command later and…success! The test pod was using the newly created storageclass that was backed by the nfs share on truenas. After that I rushed to deploy my standard set of gitea, droneci and minecraft to test it out and it was working for real without a hitch.

Conclusion

This was a long journey and it isn’t coming to a close any time soon. The next step is to continuous deployment but that’s an issue for another time. To conclude as briefly as possible, I’ve no put most of the critical pieces of my homelab in a single git repo that I can use to recreate almost all of it from scratch with minimal manual intervention (there are still a few quirks like not having dynamic inventory for ansible and having to manually copy IPs but I can put that aside for now). Despite the imperfections like not having dynamic inventory for ansible or having secrets stored unencrypted in git, I’m very happy with the setup is working. If you missed them, make sure to check out the hardware updates and the first part about software changes for a better of what I’ve been up to.

Homelab Evolved

Sun, 07 Feb 2021 13:53:10 +0200

The hardware side of the changes made to my homelab were covered in a previous post where I also alluded to some software changes.

What came before

This is the first part explaining more in-depth the issues that I had and how I dealt with them. Let’s set the stage first, my homelab was working fine in many ways. It was nice and stable debian install running kvm virtual machines using qemu and libvirt with virt-manager to perform the simpler start/stop tasks. There was really nothing wrong with it, I was exploring different software, running some services on the local network and even hosting a public website on it.

Good enough is not good enough

However as someone interested in infrastructure a few things were bothering me. Sure, I was using docker-compose with traefik for my public website and services. Yes, I was using ansible to manage the configuration of most virtual machines. But “good enough” and “I guess it works” doesn’t cut it. On the upside, I really liked having ZFS snapshots on a 2x2TB mirror. The ability to go back to a point in time where things were not broken had saved me a couple times when I inevitably broke one thing or deleted a file that could be replaced but would take a while. My backups were also on that ZFS mirror, on a different dataset of course, and that also very useful. I knew I wanted ZFS and that there was much room for improvement in regards to handling how I was running services. The single docker-compose virtual machine for the public stuff and another one for local stuff were mostly adequate but the manual management was not. Not to mention that and all my precious stuff was on one box which is well below ideal.

Taking the step

One day I decided it was time to move up in the world. Despite using docker and docker-compose for years I had never dived into kubernetes because it seemed difficult and complex. New syntax, new system with its own architecture and internal structure, new workflow, new everything. I decided to set up 3 VMs and played with default k8s as well as k3s on them. Dipping my toes into it was pretty fun and since nothing depended on it I was free to wipe it and start over. The same issue kept cropping up, having to manually install debian thrice and set up kubernetes just to delete all of it and redo it was again, less than ideal Initially I started writing a playbook before finding out that someone smarter had already gone down this path. Setting up k3s-ansible was easy so there was at least kubernetes setup automation but the whole thing was too thrown together there was no shared storage. However, I was becoming more familiar with kubernetes and learning the concepts as well as how to write yaml manifests.

Conclusion

At some point it became too bothersome, a lot of manual work was involved and basic stuff like copying over ssh keys was also done mostly manually so I had to move on. While all of this was going on, I had started planning what hardware updates I would get which you can read about here: Hardware Updates 2020