Homelab on inherent site

Home Assistant

Sat, 07 Jun 2025 14:12:34 +0300

I recently set up Home Assistant as a way of getting into home automation. It’s a really cool piece of software and took a bit of time to set up so I’d like to document everything I’ve changed here.

Getting started

Home Assistant is an open-source home automation solution, possibly the most well-known in its category. It’s a great DIY and self-hosted alternative to smart home offering from bigger vendors. As a platform it can integrate with tons of products available on the market and is very easy to run. I’ve been interested in using Home Assistant for a few years but I never had anything to use it for or with. Recently I decided I should look more into it so I ordered a SNZB-02D temperature and humidity sensor as well as a ZBdongle-E ZigBee stick also by Sonoff. ZigBee is a protocol that a lot of the smart home devices use to communicate, including my temperature/humidity monitor. This means that as long nothing goes wrong, I should be able to track my room’s temperature inside Home Assistant. Sonoff also has smart power plugs, look for the ones that have ZB in the model number like the S26R2ZB or S60ZBTPF (I didn’t check and wasted 10 euros).

I picked up the items I ordered, dusted off the old Raspberry Pi 3b+ I’ve had for years and installed Home Assistant OS on it. It took about an hour because for some reason the ethernet port on the Pi decided to not work. After facing its demons and working through the childhood neglect trauma I had given it, the ethernet blinky light blessed the little Pi. A little network setup and ta-da! I can log in using a browser on my desktop. Following that I shut it down, connected the Sonoff USB ZigBee stick and turned it back on. With a few clicks the USB stick was detected and ready to be used inside Home Assistant. I made it scan for devices and set the temperature and humidity monitor into pairing mode and then connected to it. This was a surprisingly straightforward process and right after there was data visible in the homepage.

A couple weeks later I picked up another two sensors and two zigbee power plugs. As a word of advice, the S26R2ZB is cheap so it only supports on/off functionality and not power measurement. The S60ZBTPF should support both though, I have the non-ZigBee version and it does but only using eWeLink.

Graphing the worst star’s waste heat

I keep the monitoring device on my desk so I can look at it and check that I’m not going crazy and the weather is indeed getting warmer. I also have one out in the balcony and one in the living room. After few days I could look at the pretty graphs and see how the temperature fluctuated throughout the days which was quite interesting. I could visibly see when I had opened the balcony door for some fresh air or how much the air-conditioner actually drops the temperature. In general the whole solution works very well. One day there was notification about a firmware updated for the ZigBee USB stick which I could install directly through Home Assistant. The sensors also had a firmware update and could also be updated directly through HomeAssistant. I don’t know why this seems like such a big deal to me but it’s surprising to see this process be supported and vendors not pulling some bs for once. I still hate warm weather and the sun but now I can track how much it’s affecting me.

Add-ons

For any web interface I consider it necessary to have an HTTPs certificate and some way to remotely access it. Thankfully using add-ons, it’s possible to get certs from LetsEncrypt, use nginx as a reverse proxy and also install cloudflared. Cloudflared is not official but it does work well even if it was a bit troublesome to set up initially. It does require some fiddling in the YAML configuration file (using the editor add-on) but it’s not too difficult. I also enabled prometheus metrics, which require authentication to scrape so it will be a slight bit more complicated than usual but nothing unusual. There is a lot of other software available as an add-on, many things which I don’t care to run on Home Assistant on my Raspberry Pi, but it’s good to see such robust support and that they are available for use cases where they make sense.

Later expansion

I plan to add more stuff in the future and start playing with automations. Currently it’s just a couple lights, the sensors and the power plugs. There is no inclination to do something extra with them, just having control of all of them through one place is helpful enough.

Conclusion

It’s a cool setup that works for what I want to do right now and also has a lot of room for expansion. Now that I can access it from the outside and have TLS certificates it been a very nice experience. If you’ve been waiting to start as well, it’s a lot easier than you might think.

Remote Access Homelab

Sun, 04 May 2025 18:26:25 +0300

One of the remaining things for my homelab to have is remote access. I set up a couple options and faced some issues during the process which I want to document here.

Tailscale

Tailscale is wireguard-based VPN solution. Its main idea is that you run it alongside whatever you want to be accessible in your tailscale virtual network and then you can get to it. My primary concern was replacing something like openvpn where it can be set up so it’s almost like being connected to the local network. In tailscale this can be achieved by making it announce that it can route to a specific subnet. This way things like local DNS works normally (for example I can access opnsense.home.inherently.xyzand see the web interface of my router). Additionally, you can make your traffic be routed through a specific device. This might be the router at home which will be marked as an exit node and then in the mobile app select it to be used as such. The choice of tailscale dates back to when I was still running pfSense. In pfSense there has been a built-in integration for quite a while that can work as an exit node and subnet router. In OPNSense it used to require a very manual install and setup process through the FreeBSD ports system but now there is a plugin which is a lot more user-friendly. I set it up that way and then joined my phone and my tablet to the tailnet which means I can access all of my servers remotely. Problem solved, let’s go home, roll the outro. Not yet. The thing is that while this works fine for just myself, if I want someone to check out some new selfhosted service I have or connect to a database I’m hosting I don’t want them to have to set up tailscale first.

Cloudflare Tunnel

To address that need, I chose Cloudflare Tunnel. This works a bit differently than Tailscale, you run cloudflared and tell it where traffic for a domain name should be redirected. In my case it runs inside kubernetes and points to the ingress controller. This… took a while for me to get working. It was mostly my own fault and lack of knowledge of how cloudflare works. There are certificates that cloudflare generates which encrypt traffic between clients and cloudflare. This process is done automatically and only for the root domain, for any subdomains you better get your wallet out. This meant that since I was trying to access services that were something like service.home.inherently.xyz, TLS was broken. Took about 2 days to figure this out and I only managed it thanks to Kryptonian in the Home Operationsdiscord server. The way I fixed the TLS problem was by just putting everything under the root domain which then made it work. There is a bit more work to do because I think cluster-template does have a good idea with separating services that should be externally accessible by running two ingress controllers. However, at this point it gets the job done so I’m reasonably happy with it.

Conclusion

Access do be external.

Talos Linux Setup and Configuration

Tue, 11 Mar 2025 19:28:45 +0200

After about two weeks since the first time I installed Talos, I think I’ve figured out a good setup. In the previous post I went over the PXE setup more than the kubernetes part of it so now I’ll cover the other half.

How the Talos configuration works

Following the getting started guide tells us that to generate the configuration files we need to run something like talosctl gen config homecluster https://10.0.50.69:6443. This command will generate 3 files. These are controlplane.yaml, worker.yaml and talosconfig. The control-plane nodes will use the first file, the worker nodes will use the second file and the third one should be placed into ~/.talos/config (with an edit to specify the node IP addresses or as Talos calls them in this case, endpoints). If we run the command we’ll get some output like this:

generating PKI and tokens
created controlplane.yaml
created worker.yaml
created talosconfig

Then, if we try to re-run the same command we will get a message like the following:

1
2

generating PKI and tokens
file "/home/angle/talos/controlplane.yaml" already exists, use --force to overwrite

If we do the “just go away and do it anyway” thing that it says and add --force it will again show the previous output. In this process, we’ve wiped all the secret tokens and certificates it generated before and cannot recover them. For a first-time installation this is barely an inconvenience, we probably do not care. When doing this for an existing cluster, we will also need to get a new kubeconfig since the secrets that are re-generated are also relevant for the certificate that is stored in the kubeconfig to access the cluster. That’s one of the reasons that if you’ve had the cluster running for a couple months, this might be a bit more annoying.

Separate secrets

The alternative is to take a more in-depth look at the documentation which will land us on a page with notes for production-level setups, specifically the section about separating out secrets. Following the examples in that page we can run something like talosctl gen secrets -o secrets.yaml first and then talosctl gen config --with-secrets secrets.yaml homecluster https://10.0.50.69:6443. This way, whenever we run the command to generate the configuration files (controlplane.yaml, worker.yaml, talosconfig) we will not also wipe the secrets. Now, why is that process useful or even preferable?

Using this approach, the secrets can be stored separately from the configuration itself, that’s the obvious part. The less obvious part is that given the same talosctl, the CLI tool, we can generate the same 3 files. Additionally, the kubeconfig that we can receive by running talosctl kubeconfig ~/.config/kube/config --nodes 10.0.50.72 --endpoints 10.0.50.72 is not going to need to be updated even if we reinstall Talos. You might then think, if we continually re-generate these 3 files, how can we ever permanently store configuration changes? Putting them directly in controlplane.yaml and worker.yaml is a no-go since they’ll be overwritten by a talosctl gen config command. That’s where patches come in.

Patches

The talosctl CLI tool has a couple helpful options for applying patches when generating configuration. Similar to having one configuration file for control-plane nodes and one for workers, the CLI flags are the same way. There are 3 flags: --config-patch-control-plane which applies the patch only to control-plane nodes, --config-patch-worker which applies the patch only to worker nodes and --config-patch which applies it to both. Using these, we can write patches and apply them as we wish to any and all of the nodes in our cluster. All 3 of the flags accept either inline JSON patches or YAML files depending on the notation. Now I’ll switch gears a bit and discuss my own specific setup as an example. I have 3 patches at the moment. First, is a baseline that configures network interfaces, defines the installation disk, sets up NTP, adds kernel parameters, enables workload scheduling on the control-plane nodes and certificate rotation for the kubelet. The other two are specific for the openebs storage system and for the metallb bare-metal loadbalancer. It’s important to note at this point that it merges the changes. So if you want to remove something from the default configuration (such as the node.kubernetes.io/exclude-from-external-load-balancers: "" node label) it’s not as easy as just having the key specified but being empty (for example nodeLabels: {}). To generate my configuration, I can run the following:

talosctl gen config homecluster https://10.0.50.69:6443 \
	--with-examples=false \
	--with-docs=false \
	--with-secrets=secrets.yaml \
	--config-patch=@patches/01_baseline-changes-patch.yaml \
	--config-patch=@patches/02_openebs-mayastor-patch.yaml \
	--config-patch=@patches/03_metallb-patch.yaml \
	--endpoints=10.0.50.71,10.0.50.72,10.0.50.73

Here I’ve disabled documentation and examples just so the file isn’t as cluttered. The important part though is that the sensitive information is only stored in only one file which could be encrypted and put into version control. From there, I could use the same version of the CLI tool to generate the final configuration file. The goal is to be able to have some sort of GitOPS setup where all or most of what’s needed to create the Talos configuration is stored in the same infra repository as everything else. One tool that seems to fit this exact use-case is talhelper but I’ve not looked into it yet.

Conclusion

I keep finding cool stuff about Talos and the homelab tinkering will continue until morale improves.

PXE setup for Talos Linux with Matchbox

Sat, 01 Mar 2025 19:00:33 +0200

Join my campaign of hate against installing operating systems interactively. I usually wipe my homelab virtualization hosts every week or two. This might sound insane but I’ve been working on automating their setup. As explained in the previous post, friendship ended with ProxMox and kubernetes will take its place. Specifically Talos Linux, a linux distro you can’t even SSH into. ProxMox kind of notoriously doesn’t have a way to be sanely installed through PXE but Talos is kind of made for it. However, PXE (and iPXE, we’ll discuss this later) is not perfectly documented and can be even harder to debug. So in this post I’ll document what worked for me and talk about some pretty helpful resources.

Plan

PXE

So let’s start with a bit of background, then lay out what I wanted to achieve, why and what I considered “good enough”. First off, PXE. It stands for pre-boot execution environment and is mostly reliant on motherboards and network cards to be implemented on the client side. It’s a stripped down environment before the computer is booted into an operating system. Inside it, there are standards for getting information about what to execute, where to get it and what configuration to use. PXE is the name of the general concept and the first version of the technology. It allows booting a computer from the network by loading files from a TFTP server. Later on, iPXE came around to fix some of its limitations. One thing it’s useful for having a computer and being able to boot it from the network and install an operating system on it without having to have a USB stick on you. With something like syslinux you can even create menus to be able to choose what to install and have a bunch of different options with different settings. As computer installations grow larger, it’s easy to see how this becomes kind of essential as you can’t run around multiple datacenters any time something needs to be re-imaged. A counter-point to that could be that servers have remote management features allowing us to load ISOs as if they’re DVDs but even then, iPXE has an advantage because you can fully automate the process. Also my poor version of servers (read: used office PCs) do not have IPMI. At any rate, that’s the general idea of PXE.

Talos

How does that apply to my case then? I want to be able to completely wipe all three systems and then effortlessly restore them. As I plan to run kubernetes on it, the installation of applications to the cluster is taken care of using FluxCD so I only really need to solve the installation part. That’s where Talos comes in. Talos is an incredibly minimal linux distribution that is designed with the sole purpose of running kubernetes. No SSH, no TTY, no nothing. Sitting in front of the system only allows you to change kernel boot parameters and the network configuration. Everyhing is done using the API which is accessible interactively through the talosctl CLI utility. This means that we can have our operating system and kubernetes configuration as code for the price of one config file. Sign me up!

Matchbox

Those are the ingredients but it can hardly be called a meal. We need to get some kitchen utensils in this analogy to get the desired result. Our PXE setup will need to have a little bit of logic in it so it knows what to install with what configuration. See, Talos can be told to fetch a configuration file from an HTTP server using a kernel parameter. A conditionally-rendered file would allow us to alter the response depending on who’s requesting it. This is where matchbox slots in. When matchbox gets a request it can see some information about the host requesting it and therefore make a determination about what to reply with. We can then serve different options according to which system is asking such as what Talos configuration file to use. This way the control-plane kubernetes nodes can receive the Talos controlplane.yaml while worker nodes get the worker.yaml. So a brand-new host with an empty disk drives would boot up, skip the disks since they don’t have an operating system and then try booting using PXE. Then the host would get its configuration, install the operating system, install kubernetes and become part of a cluster.

Definition of Done

The ideal process when getting adding a host should look like:

note down the MAC address and corresponding IP somewhere
plug in the cables
configure the boot device order
save settings and exit
play minecraft while waiting
run one or zero commands
host shows up in kubectl get nodes
recreating it from scratch should be just as easy

I don’t know if I can get exactly to that point but I’m willing to put in some effort into the setup so in the future I can avoid it to a greater extent. Some compromise is acceptable since I don’t have any IPAM infrastructure in place already so this isn’t the full and proper way to go about it.

Implementation

How do I do X in Y minutes

Meat and potatoes first, for quick reference here is what I did (5-6 hours of suffering omitted for brevity):

Installed tftp-hpa on Arch
Downloaded undionly.kpxe and ipxe.efi
Put them in /srv/tftp

Created matchbox.ipxe with the following contents:

1
2

#!ipxe
chain http://10.0.20.50:8080/boot.ipxe

Installed matchbox using Docker by running the following command:

`1`	`docker run --net=host --rm -v /var/lib/matchbox:/var/lib/matchbox:Z -v /etc/matchbox:/etc/matchbox:Z,ro quay.io/poseidon/matchbox:v0.10.0 -address=0.0.0.0:8080 -log-level=debug`

Downloaded vmlinuz-amd64 and initramfs-amd64.xz
Put them in /var/lib/matchbox/assets
Download talosctl
Run talosctl gen config homecluster https://10.0.50.69
Edit controlplane.yaml until it looks good
Put controlplane.yaml in /var/lib/matchbox/assets as well

Create /var/lib/matchbox/profiles/control-plane.json:

{
  "id": "control-plane",
  "name": "control-plane",
  "boot": {
    "kernel": "/assets/vmlinuz",
    "initrd": ["/assets/initramfs.xz"],
    "args": [
      "initrd=initramfs.xz",
      "init_on_alloc=1",
      "slab_nomerge",
      "pti=on",
      "console=tty0",
      "printk.devkmsg=on",
      "talos.platform=metal",
      "talos.config=http://10.0.20.50:8080/assets/controlplane.yaml"
    ]
  }
}

Create /var/lib/matchbox/groups/control-plane.json:

{
  "id": "kube01",
  "name": "kube01",
  "profile": "control-plane",
  "selector": {
    "mac": "18:03:73:2a:d8:a2"
  }
}

In OPNSense (link to my router)
- Under DHCP Static Mappings for this interface. add the hosts
- Under TFTP server:
  - Set TFTP hostname: 10.0.20.50
  - Set Bootfile: undionly.kpxe
- Under Network booting
  - Set next-server IP: 10.0.20.50
  - Set default bios filename: undionly.kpxe
  - Set iPXE boot filename: matchbox.ipxe
Boot Optiplex
Wait
Watch text scroll on screen
???
No profit, I spent my whole weekend on this

Explanation

PXE is an under-documented part of the common network services one can set up. An amusing yet educational thing I found was a redditor’s 3-part descent into madness and consequently enlightenment:

The matchbox documentation was also very good at explaining both the general process and giving some specific recommendations:

With those out of the way I’ll make an attemt at explaining it myself. When a PXE client starts, it does the DHCP dance which tells it where the TFTP server is and what the name of the file it should execute is. That file is a network boot program which can optionally load some configuration (also stored on the TFTP server). So after all of that is figured out, the PXE client will pull the kernel and initramfs, again from the TFTP server. In contrast, iPXE uses scripts stored on the TFTP server instead of plain configuration files so we can do fancier things with it. However, my old desktop PCs do not have iPXE so I had to use an in-between thingy, undionly.kpxe. What that does is load iPXE firmware (ipxe.efi) so we can access the cool new features without having to replace any hardware. When that loads, it talks to the DHCP server as an iPXE client this time and gets told to grab the matchbox.ipxe file from the TFTP server. That file is an iPXE script which tells the client to check the matchbox server. When matchbox gets the request, it checks the groups it has registered and if there are any matches according to the selector field, it uses the corresponding profile. In this case, the profile is for control-plane Talos Linux nodes. The kernel and initramfs listed in the profile are used alongside some specific kernel parameters. The most important parameter is talos.config=http://10.0.20.50:8080/assets/controlplane.yaml which tells the machine that is booting the operating system to fetch the configuration file listed there. Of course, you will need to have generated the configuration (using talosctl gen config homecluster https://10.0.50.69:6443) and moved it in the matchbox assets directory (or in another HTTP server, matchbox is just convenient in this setup). That configuration has some basic things like which disk and network settings to use. When the machine gets this configuration, it installs Talos and then waits. At this point the system is booted normally, no PXE involved.

Following that, all we need to do is run talosctl bootstrap --nodes 10.0.50.71 --endpoints 10.0.50.71 and we get a kubernetes cluster. To get the kubeconfig required to access the cluster, run talosctl kubeconfig my-talos-kubeconfig --nodes 10.0.50.71 --endpoints 10.0.50.71 and then check that it works by running kubectl --kubeconfig my-talos-kubeconfig get nodes. I’ve tested the whole process end-to-end twice now to ensure that nodes with blank disks and correct boot order will behave correctly and it’s almost magic. True enough, after pressing the power button on all three systems, making some coffee, and then running the commands mentioned previously, I got this output:

$ kubectl get nodes -o wide
NAME     STATUS   ROLES           AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION   CONTAINER-RUNTIME
kube01   Ready    control-plane   3h20m   v1.32.2   10.0.50.71    <none>        Talos (v1.9.4)   6.12.13-talos    containerd://2.0.2
kube02   Ready    control-plane   3h20m   v1.32.2   10.0.50.72    <none>        Talos (v1.9.4)   6.12.13-talos    containerd://2.0.2
kube03   Ready    control-plane   3h20m   v1.32.2   10.0.50.73    <none>        Talos (v1.9.4)   6.12.13-talos    containerd://2.0.2

And I guess that’s it. It needed some effort and research and the approach should be adjusted according to the network situation (the matchbox docs saved the day here). In the end though, I really like what I got working and it was a fun way to spend the weekend. I can go from basically brand new computer with nothing on it to full cluster with a high availability control-plane using patience and a single command. Two commands and I can install fluxcd on the cluster which will install all my applications. Looking back at the definition of done I set in the beginning, I consider this a success.

Conclusion

As the proud owner of a Talos Linux kubernetes cluster installed in a fully automated way, I am quite happy. I can get back into running services at home and live through the pain of using kubernetes daily again. I have a couple of ideas in mind for how to use my new container running platform but that’s for another post. Hopefully you got a better idea of what PXE is, how it works and how I used it in this instance. If you weren’t familiar with any of this, maybe it’s time to get your hands dirty and prepare to throw away your USB drives. Either way, that’s all for now.

Homelab 2025 Plans

Fri, 28 Feb 2025 22:51:33 +0200

Intro

After recapping the state of my homelab and writing down how it was in the end of 2024, I think it’s time to look towards the future. Having finally finished my thesis, I now have less constant stress to deal with and a bit more time than I’ve had in the last 2 and a half years. The main goal of rebuilding my homelab is to make it an actual production environment. This means that a few important problems need to be solved before it’s considered done. Along with that, I’m willing to lose a bit of flexibility in exchange for having something that actually works which is a bit odd for me to write but alas.

Hardware

Over the past several years I’ve invested quite a bit into the hardware that already exists. I won’t go into great depth about it, all the details are in the previous post. My homelab is housed in a 27U rack, includes a managed switch, a DIY router, 3 virtualization hosts and at least one NAS. That is more than enough to at least get started running several services. I will try to avoid investing in hardware this year with two exceptions, a VGA KVM and a good NIC for the router. Other than those two additions, I plan to focus on software choice and configuration.

Software

Kubernetes

I’m going bare metal kubernetes. This might seem counter-intuitive but it’s actually the easiest thing to get working. I still like ProxMox, I’m happy with the terraform provider for working with ProxMox and there is now a healthy alternative one, I use ansible at work and it’s all nice but I’m looking for an even higher level of automation. The integrations available for kubernetes are at a great point and have been for a couple of years now. I can forego dealing with issues such as “how do I make DNS automatically update when I start running a new service” and focus more on running and using my self-hosted services. Of course, kubernetes presents its own set of challenges and has its own quirks but I believe the tradeoff to be worth it for me.

The plan right now is to go with Talos Linux so my hosts are nice and lean. The entire process of installation can be automated using the matchbox PXE server and a bit of configuration and the operating system has just enough to run kubernetes and that’s it. I could of course regret this in which case I’ll go back to Debian my beloved but it remains to be seen. Initially I’ll go with a simple ISO installation to get started and then I’ll see if I want to take it a step further. The kubernetes-related stuff is mostly figured out, default CNI, nginx for ingress, metallb for external IPs, flux to deploy stuff and I’ll try adding renovate bot to help with maintenance. What remains is to decide if I’m going to go with openebs or longhorn for storage but that shouldn’t be too difficult or time-consuming.

Router

The other interesting software choice is the router. Having a dedicated machine for it seems wasteful and I’m bound to running whatever software is available for FreeBSD. No, I’m not planning on moving away from OPNSense at the moment but I do want to start the vicious cycle again. The plan is to have it in a VM, thus the vicious cycle, and then be able to run things like matchbox on the same machine, a different NTP server, a different DHCP or PXE server (mainly tinkerbell comes to mind), different DNS server (Hickory seems quite interesting) and so on. Those could be containers inside other VMs or whatever else, I’m not bound to whatever is packaged for FreeBSD or even specifically for OPNSense. This of course requires a hypervisor (ProxMox makes a comeback) and a NIC that can be passed through to a virtual machine in order to run the router. This is a secondary concern since I do have an old machine from the core2duo days if I just want to runs some basic network infrastructure like the ones mentioned previously for experimentation purposes. VyOS has also been on my mind but that’s a bigger jump and would require quite a bit of effort to adapt everything I have to it. The router plans are unclear at the moment so it’s more of a backburner project.

Conclusion

One must imagine Sisyphus, he rolls the rock and I rebuild my homelab. Was there a point to writing this? I’m not sure. These are my plans at the moment so I can get back into the hobby. If I do go through with them, I’ll try to document the experience when I have something workable.

State of Homelab 2024

Mon, 27 Jan 2025 22:38:48 +0200

When operating a homelab there is always something new to do or something waiting to be fixed. This post’s aim is to detail the state of my homelab at the end of 2024 and beginning of 2025.

Router

The previous post was about virtualizing pfSense inside ProxMox but this setup is no longer in use. In its place, there is a new system running OPNSense on bare-metal. This was done for a few reasons, the main one being that I wanted to move away from pfSense. More than that, there was a circular dependency where the ProxMox host had to be up in order for the router to function but in order for the cluster nodes to talk to each other, the router needed to be online. In practice this was not a huge issue but it bothered me since it made experimenting more difficult because recreating the cluster meant taking down the router as well. I’ve lost a few things, mainly pfblocker-ng and native tailscale integration but they’re not as big of a loss as I had anticipated. In the end I think it was worth it, over the course of the past few years the company making pfSense has acted against the best interest of pfSense users and I did not feel like using their software anymore.

Virtualization

I’m still running ProxMox, although now it is a 3-node cluster. I’ve spent a pretty significant amount of time tweaking the bootstrap process from a fresh install to having the entire homelab operational. Somewhere along the line I tried making it so non-cluster operation of multiple hosts was supported but gave up on that eventually. Each of the nodes has a 250GB SSD used as a boot drive and for node-local storage along with a 2TB HDD used as a Ceph OSD. Since the last post about using ansible and terraform with ProxMox, I’ve figured out the process of adding tools to the debian image in the virtual machine template. There is also the option to deploy using GitLab CI so just making committing and pushing to the repo is enough to create a new virtual machine. The k3s virtual machine terraform configuration was improved significantly but I’ve not adopted modules yet which seems to be the next step for organization-related improvements. I also run all software directly on top of debian virtual machines without containers to practice writing ansible. At work, our team runs everything in Docker so it could be a good idea to start moving to something similar at home for the sake of practice.

Kubernetes

The kubernetes cluster got a significant update. I’ve adopted FluxCD to a greater degree, figured out secrets storage and a pretty good structure for the YAML files. The k3s cluster has 3 control plane nodes and 3 worker nodes, one per virtualization host. I’m using longhorn for storage with plans to try and use the Ceph cluster as an additional storage class. The networking setup consists of MetalLB to have an IP in the subnet and then run ingress-nginx as the ingress controller on that IP. No plans for checking out Gateway API have been made at current but it is a good and necessary step forward in the ecosystem, I plan to move to it after things are a bit less in flux. The DNS is manual right now due to the low amount of services but I’ve tested external-dns with PiHole and I’m aware of the OPNSense webhook option, it remains to be seen what I’ll end up using. The setup needs a bit more tender love and care but it’s running well and doing what I need it to. I have plans to move more stuff to the cluster after the foundation is solid and not just have my own demo applications in there. The thought has crossed my mind to completely do away with ProxMox and then run Talos Linux bare metal on the systems but I’d be losing out on some flexibility which I don’t want to commit to right now. All in all, kubernetes is still pretty fun and the 1.30 and later releases have made good quality of life improvements.

NAS

Since I upgraded my desktop, the old one’s internals were transplanted to a different case are now a second NAS. I’m saving up to get a big external drive so I can have an extra copy of the current main NAS so I can freely mess with them to have a solid backup solution. The changes in TrueNAS Scale regarding applications has left me waiting for things to settle down before I commit to a final setup. Overall nothing exciting on this front.

Hardware

I run mostly used office desktops and a basic managed switch. I have 3 Dell Optiplex machines: 2nd gen, 3rd gen and 4th gen Intel i5 CPUs with RAM sizes ranging from 16 to 32GB each. For storage they have a 250GB SSD and a 2TB HDD as mentioned earlier. The router is a Fujitsu equivalent desktop with a 4th gen Intel Pentium and 8GB of RAM as well as a 500GB SSD. My rackmount NAS is an i5 4570 with 16GB of RAM and 4x4TB HDDs in a 3-drive ZFS RAID-Z1 plus one hot spare. My backup NAS is an i7 4770 with 32GB of RAM and 2x8TB HDDs in a 2-way ZFS mirror. The main network switch is a Netgear GS724TV4, just a layer 2 managed gigabit switch.

Conclusion

I wanted to write all of this down to have a reference for myself when I inevitably change something and it might help someone else as well. Also the blog is back(?) after more than a 2 year silence, hopefully I will not neglect it for that long again.

QEMU Guest Agent for Terraform on ProxMox

Wed, 27 Jul 2022 13:38:18 +0300

In a previous post I talked about how I set up proxmox in order to be able to provision virtual machines on it using terraform. Since then, I’ve improved the setup by adding qemu-guest-agent to the template images without needing to create them myself from scratch and I’d like to share it here.

The Problem

A lot images intended for openstack or cloud use, be they debian or ubuntu or something else, do not include qemu-guest-agent by default. This means that we either don’t get to enable that integration or have to do a complex dance of provisioning the VMs with the agent disabled, installing qemu-guest-agent, turning them off, applying a terraform plan with the agent enabled and booting them back up. Obviously that’s very tedious, error-prone and time-consuming especially when creating and destroying potentially dozens of VMs for testing. To figure out a solution, I went down a bit of a rabbit hole and considered a few solutions before settling on one so I’d like to take you through the selection process.

The Solutions

There are multiple ways to solve this problem and there isn’t really a standard go-to recommended one. One of these might work better for your situation and that’s great however I will be detailing why I didn’t choose them.

Images with qemu-guest-agent pre-installed

This was my first instinct, just find images that have it already. Unfortunately I couldn’t find a debian image that fit this criteria and overall it’s not a reliable solution, some distros might simply elect to not include it.

Terraform remote-exec provisioner

I found this great blog post by Besmir Zanaj which was really interesting and demonstrates how to use terraform to achieve post-provision VM customization. It is worth checking out however for something so basic that all virtual machines need I decided it was better to bake it into the template rather than have the action run on each VM individually. The example from the post about firewall rules is a great use-case for this functionality so it’s not to be discarded entirely.

Packer proxmox-iso Builder

Packer is another tool from Hashicorp, the same company behind terraform, that can create images to be used with proxmox and then make templates based on those images. The functionality is offered by a plugin, specifically a builder, called proxmox-iso. It didn’t appear to be a terrible choice but the configuration examples that I found relied on launching ubuntu, waiting a certain amount of time then sending keystrokes to begin a no-cloud autoinstall. This seemed to me to be very error-prone and distro-specific so I skipped it. I don’t believe it to be a complete dead-end and it’s possible there is a better way to use it for what I want but I couldn’t figure out how.

Adding qemu-guest-agent using guestfs-tools

The last thing I looked into was using guestfs-tools to modify the image before creating a proxmox template out of it. Guest-fs tools is a set of tools for modifying virtual machine disk images that works on a lot of distros. After lots of searching, I came across an awesome blog post by Austin Pivarnik which tackled basically the same problem. As detailed in the post itself, on debian-based systems which luckily proxmox is, we can install libguestfs-tools and then use the virt-customize command to alter the contents of a virtual machine image. How I added this to my setup:

add libguestfs-tools to the list of packages installed by my proxmox setup ansible playbook
add a task (adapted for all VM images) to the same playbook after downloading the image but before creating the template
run playbook
ta-da! You can see the changes I made in this commit where I also adjusted my terraform configuration and a couple other small things. I’ve been very happy with this so far and aside from showing the guest IP in the proxmox web interface as well as allowing smooth shutdown it has also improved provisioning times.

Conclusion

Having good templates for virtual machines on a virtualization platform is very important for automation. While it took quite a long time for me to find a suitable solution I am quite happy with where I ended up and the improvement to my homelab. Thank you for reading and I hope you learned something new.

Network Design Document

Sat, 16 Jul 2022 18:45:44 +0300

I recently started remaking my network at home and decided to publish some of the design documentation here

Network Layout

VDSL Internet comes in through an RJ11 port to a Vodafone H300s modem-router. LAN port 2 of the H300s is plugged into port 2 (untagged VLAN 10) of the Netgear GS724Tv4 switch. Switch port 23 (untagged VLAN 1, tagged in all other VLANs) is plugged into the gigabit ethernet port of the pfsense router. All devices are plugged directly into the switch since they fit. Inter-VLAN routing happens through pfsense and the switch operates as a managed L2 switch even though it has L3 capabilities.

VLANs

Probably overkill but certainly enough:

VLAN	Description	Subnet	Gateway	DHCP range from pfsense
1	Default	-		-
2	Auto VoIP	-	-	-
3	Auto-Video	-	-	-
10	Border Router LAN	10.0.10.0/24	10.0.10.1	-
20	End Devices	10.0.20.0/24	10.0.20.254	10.0.20.10-245
22	Trusted	10.0.20.0/24	10.0.22.254	10.0.22.22-222
30	Monitoring	10.0.30.0/24	10.0.30.254	-
40	Storage	10.0.40.0/24	10.0.40.254	-
50	Servers	10.0.50.0/24	10.0.50.254	-
60	Guest	10.0.60.0/24	10.0.60.254	10.0.60.2-252
65	Media	10.0.65.0/24	10.0.65.254	10.0.65.2-252
66	IoT	10.0.66.0/24	10.0.66.254	10.0.66.2-252
67	DIY IoT	10.0.67.0/24	10.0.67.254	10.0.67.2-252
70	Security	10.0.70.0/24	10.0.70.254	-
80	DMZ	10.0.80.0/24	10.0.80.254	-
90	APIs	10.0.90.0/24	10.0.90.254	-
99	Management	10.99.0.0/16	10.99.0.1	-
111	Kubernetes	10.0.111.0/24	10.0.111.254	-
113	MAAS	10.0.113.0/24	10.0.113.254	-
123	Windows	10.0.123.0/24	10.0.123.254	-
150	Test LAN	-	-	-
666	Native	-	-	-
1022	Unused Port	-	-	-
1337	Danger	-	-	-

Conclusion

That’s my home LAN currently, I’ll try to keep this updated as I change stuff but no promises.

Ansible, Terraform and Proxmox

Fri, 24 Jun 2022 23:24:57 +0300

In the continuing quest to improve my homelab I’ve written a few ansible playbooks as well as terraform configuration to automate a lot of tasks. The initial setup of proxmox is handled by ansible, the provisioning of virtual machines is handled by terraform and the installation of k3s on the VMs is done by ansible.

Proxmox Setup with Ansible

Before getting to use all the shiny devops with a selfhosted solution, we need to do a bit of setup first. This isn’t perfect since it’s just ansible running a bash script but it gets the job done. The other issue,, is that a lot of cloud images don’t include qemu-guest-agent and other packages by default. I have a few ideas how to solve that (check out this TODO.md for more info) but haven’t tested out anything yet. Since qemu-guest-agent and things like nfs-common are required on most VMs, a lot of redundant package installation happens but such is life for now. With that out of the way, let’s start with how I configure proxmox.

Host

While proxmox has a lot of things ready to go out of the box, a bit of configuration is always required and even more so in this case. The goal of the following steps is to turn our hosts into capable, cloud-like, targets for provisioning resources using terraform.

Repository

Initially the enterprise repository which we don’t have a license to access is enabled. To fix this, I delete the file in /etc/apt/sources.list.d that contains it and add a file with the no-subscription repository. This might not be necessary since the lae.proxmox ansible role can do essentially the same thing and many more. Here are the ansible tasks:

    - name: remove enterprise repo
      file:
        path: /etc/apt/sources.list.d/pve-enterprise.list
        state: absent

    - name: add no-subscription repo
      copy:
        src: pve-no-sub.list
        dest: /etc/apt/sources.list.d/pve-no-subscription.list
        mode: '0644'
        owner: root
        group: root

Template VM

The terraform provider for proxmox can work with cloud-init which is a great vendor-agnostic way of configuring the basics (username, password, IP addressing, DNS, SSH keys) in a virtualized environment. This is achieved by having a virtual disk or virtual cd-rom that mounts inside the VM and provides this information to the guest.

VM Image

Since software is not always compatible with the latest version of a linux distribution, in this case debian, I keep the last two versions around. The first step is to download the images:

    - name: get debian 10 cloud image
      get_url:
        url: https://cloud.debian.org/images/cloud/buster/20211011-792/debian-10-generic-amd64-20211011-792.qcow2
        dest: /root/debian-10-openstack-amd64.qcow2
        checksum: sha512:f3dac13104b4e28eb62c46cbbb1e30fc9c792834500f9101d477c19c258c31ff04850933ee0b3e63236eca38c854447d95a0ab45163c4a3fccf05f9dd95601a5
        mode: '0644'
        owner: root
        group: root

    - name: get debian 11 cloud image
      get_url:
        url: https://cloud.debian.org/images/cloud/bullseye/20220121-894/debian-11-generic-amd64-20220121-894.qcow2
        dest: /root/debian-11-openstack-amd64.qcow2
        checksum: sha512:0948dc56b4834a7755e4eae7d5532e138b90484a949161ffe9fc6894c7a14b1bd32ebf96fa3f3d03d498fe7ee125f8014f31bfab5825915a93de9330df814f7b
        mode: '0644'
        owner: root
        group: root

Template Script

As mentioned in the beginning I’d like to automate this with ansible but this is what I have for now:

if [ -z "$1" ]; then
    VMID=9001
else
    VMID="$1"
fi

if [ -z "$2" ]; then
    VMNAME="debian-tmpl"
else
    VMNAME="$2"
fi

VMEXISTS="$(qm list | grep $VMID)"

if [ "$VMEXISTS" ]; then
    echo "VM with ${VMID} already exists"
else
    qm create "${VMID}" -name "${VMNAME}" -memory 1024 -net0 virtio,bridge=vmbr0 -cores 1 -sockets 1 -cpu cputype=kvm64 -description "debbie image" -kvm 1 -numa 1
    qm importdisk "${VMID}" debian-10-openstack-amd64.qcow2 local-zfs
    qm set "${VMID}" -scsihw virtio-scsi-pci -virtio0 local-zfs:vm-"${VMID}"-disk-0
    qm set "${VMID}" -serial0 socket
    qm set "${VMID}" -boot c -bootdisk virtio0
#   qm set "${VMID}" -agent 1 #disabled since VM does not initially have qga installed
    qm set "${VMID}" -hotplug disk,network,usb,memory,cpu
    qm set "${VMID}" -vcpus 1
    qm set "${VMID}" -vga qxl
    qm set "${VMID}" -name "${VMNAME}"
    qm set "${VMID}" -ide2 local-zfs:cloudinit
    qm template "${VMID}"
fi

This works well enough for first-time setup and with a few additions could become idempotent. The above script and its debian 11 equivalent are copied over to the proxmox hosts and executed by the following ansible tasks:

    - name: get debian 10 template vm script
      copy:
        src: setup-debian-10-cloudinit.sh
        dest: /root/setup-debian-10-template.sh
        mode: '0755'
        owner: root
        group: root

    - name: run debian 10 template vm script
      shell:
        cmd: /root/setup-debian-10-template.sh

    - name: get debian 11 template vm script
      copy:
        src: setup-debian-11-cloudinit.sh
        dest: /root/setup-debian-11-template.sh
        mode: '0755'
        owner: root
        group: root

    - name: run debian 11 template vm script
      shell:
        cmd: /root/setup-debian-11-template.sh

Terraform Setup

With proxmox ready, let’s move on to the terraform setup. If you haven’t used it before, terraform is a declarative way of describing our infrastructure. It can manage virtual machines, DNS records, object storage buckets, kubernetes resources and many more things with the appropriate providers.

Provider

Providers in terraform are plugins that enable terraform to manage resources that it doesn’t support out of the box. In this case, we want to manage proxmox virtual machines and containers but it could be other things like a router running pfsense or vyos. In previous versions, the provider would need to be installed manually but nowadays we only need to create a main.tf with the provider listed and upon running terraform init, the provider will be downloaded automatically.

Basic `main.tf`

Let’s create that file then. Open your favorite editor, paste the following and replace the IP as well as the password with the ones applicable to your setup:

terraform {
	required_providers {
		proxmox = {
			source = "Telmate/proxmox"
			version = ">=2.9.5"
		}
	}
}

provider "proxmox" {
	alias = "pve"
	pm_tls_insecure = true
	pm_api_url = "https://192.168.70.1:8006/api2/json" #Replace IP
	pm_password = "password123" #Replace password
	pm_user = "root@pam"
}

Save the file and then run terraform init. You should see the following output (the version might be different):

Initializing the backend...

Initializing provider plugins...
- Finding telmate/proxmox versions matching ">= 2.9.5"...
- Installing telmate/proxmox v2.9.10...
- Installed telmate/proxmox v2.9.10 (self-signed, key ID A9EBBE091B35AFCE)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

And ta-da! We’ve installed the provider and are ready to start creating resources.

Creating a test VM

The above file won’t do anything so we’ll declare a virtual machine and then tell terraform to create it. The example below is specific to my setup so you probably want to modify it before using but here it is:

terraform {
	required_providers {
		proxmox = {
			source = "Telmate/proxmox"
			version = ">=2.9.5"
		}
	}
}

provider "proxmox" {
	alias = "pve"
	pm_tls_insecure = true
	pm_api_url = "https://192.168.70.1:8006/api2/json" #Replace IP
	pm_password = "password123" #Replace password
	pm_user = "root@pam"
}

resource "proxmox_vm_qemu" "proxmox_test_vm" {
	provider = proxmox.pve
	name = "deb-10-test"
	target_node = "pve"

	/* change to debian-templ for 11 */
	clone = "debian-tmpl"
	os_type = "cloud-init"
	cores = 2
	sockets = 1
	cpu = "host"
	memory = 2048
	scsihw = "virtio-scsi-pci"
	bootdisk = "virtio0"
	agent = 0

	disk {
		size = "30G"
		type = "virtio"
		storage = "local-zfs"
	}

	network {
		model = "virtio"
		bridge = "vmbr0"
	}

	ipconfig0 = "ip=192.168.50.25/16,gw=192.168.0.1"

	/*workstation SSH key*/
	sshkeys = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5jzKi37jm3517bqThbw+7LR/GXm3qC6Az5F+ZUa36vYM7Ygk2K5bWcFIL2YUCrkL5jfSsvoowONjCAxyuoyxtW4MJxnQLyq4u4yDsRC7YvBPAKZUYaHwnbkCfDs5a75dEFOoDxCA0DY2GrhqzBndaTcCfl0fZ4vN+9LcKOb1dSKiHeHvsh35YNtwntbL21meo+hiycUEgGwNe9/4kxKpdGTr7HvbeX2Fjm/UZBZIJKVcGop/3gCHXYnKH+OY5zc8cmt9Jg4CIwEqrSKeOX0bE8LSPRpVRXH4v8OcMaMei/HQejlH8NBwybEdJ4mhl8vHaFEjDbIWoOujmiRQF2263 angle@puddle"

	/*required otherwise the state always appears modified*/
	lifecycle {
		ignore_changes = [
			cipassword,
			network,
			desc,
		]
	}
}

After saving the file, we can run terraform plan -out create-test-vm.plan to calculate the changes required and save them to a file. Following that, we can run terraform apply create-test-vm.plan to apply the changes to our infrastructure. Some output will appear while the template is cloned, the VM settings are adjusted and the cloud-init configuration is put into the drive. In the end, you should have a brand new VM created by terraform!

Creating VMs for k3s

The above example is realistic but what if we need 5 similar VMs that could be used as kubernetes nodes? Conveniently, terraform provides a count variable that we can use to slightly differ settings where needed. Here is the resource that provisions the virtual machines that I run a k3s cluster on:

resource "proxmox_vm_qemu" "proxmox_vm_k3s" {
	provider = proxmox.pve
	count = 5
	name = "deb-k3s-${count.index + 1}"
	target_node = "pve"

	clone = "debian-tmpl"
	os_type = "cloud-init"
	cores = 2
	sockets = 1
	cpu = "host"
	memory = 3084
	scsihw = "virtio-scsi-pci"
	bootdisk = "virtio0"
	agent = 0
	onboot = true

	disk {
		size = "30G"
		type = "virtio"
		storage = "local-zfs"
	}

	network {
		model = "virtio"
		bridge = "vmbr0"
	}

	ipconfig0 = "ip=192.168.15.5${count.index + 1}/16,gw=192.168.0.1"

	sshkeys = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5jzKi37jm3517bqThbw+7LR/GXm3qC6Az5F+ZUa36vYM7Ygk2K5bWcFIL2YUCrkL5jfSsvoowONjCAxyuoyxtW4MJxnQLyq4u4yDsRC7YvBPAKZUYaHwnbkCfDs5a75dEFOoDxCA0DY2GrhqzBndaTcCfl0fZ4vN+9LcKOb1dSKiHeHvsh35YNtwntbL21meo+hiycUEgGwNe9/4kxKpdGTr7HvbeX2Fjm/UZBZIJKVcGop/3gCHXYnKH+OY5zc8cmt9Jg4CIwEqrSKeOX0bE8LSPRpVRXH4v8OcMaMei/HQejlH8NBwybEdJ4mhl8vHaFEjDbIWoOujmiRQF2263 angle@puddle"

	lifecycle {
		ignore_changes = [
			cipassword,
			network,
			desc,
		]
	}
}

There aren’t many changes from the previous example, the name and the IP address use the index in order to avoid 5 VMs having the same name and IP address and that’s it!

Conclusion

Hopefully this gives you inspiration for some awesome things you can do to automate your infrastructure and make working in a virtualized environment easier. Thank you for reading.

Cisco SR2024C Repair

Mon, 28 Mar 2022 18:03:46 +0300

One of my friends at university gave me a broken switch that seemed broken and I decided to fix it. Join me on a pretty entertaining troubleshooting path that was not entirely necessary.

Backstory

This semester in university I am taking a CCNA class. The learning path is split across two semesters, next semester I’ll take CCNA 2 to be ready to get the certification. A few of my friends are also there an one of them happened to have a supposedly broken Cisco switch. Great opportunity to use it for hands-on learning, I thought. He said it wasn’t fully broken, it could turn on fine but would turn off after 10-20 minutes. Not that bad. A little fan was also missing from it but that would be no more than 5-10 euros to replace. He gave it to me one day, I took it home and that was that. After a bit of research I realized the switch wasn’t managed meaning it didn’t have Cisco iOS capabilities. Nevertheless, I like tech necromancy and network gear seldom goes unused. So that’s basically how I got a Cisco SR2024C.

Initial Findings

After checking that the power plug looked fine and the ports didn’t look damaged, I decided to turn it on. Indeed, after roughly 20 minutes it started boot-looping so the behavior was as described. The switch wouldn’t completely shut off, instead it followed a little pattern:

lights flash for a second the way they do when first plugged in
lights go out same as when normally booting
goto step 1

This meant that eventually anything plugged into it would drop off the network which is not good. At this point I’m suspecting the power supply since the device is pretty old, about 7 years past its End-Of-Life date.

Open Wide

Behavior confirmed and reproducible, we have a place to start from. The device would need to be opened anyway in order to replace the fan and I’d like to inspect the inside too. This specific switch is considered compact as noted by the ending C in its model number (SR2024C). Its width is a bit more than the width of the ports, 24 RJ45 gigabit ethernet ports and two miniGBIC. It has holes to attach rackmount ears if desired but I wasn’t provided those. On the front there is a plastic cover that fits over the 4 sets of ports (3 sets of 8xRJ45 and one set of the 2 miniGBIC) and also clips on to the top and bottom metal pieces. I dinged the plastic since I used a pair of scissors to pry it off but that’s alright, we’re going for functional not pretty. Following that, I had to carefully unhook some little metal tabs that held the top half of the metal case to the bottom half. Push a little here, pull a little there and ta-da! We’re in. The metal tabs were annoying to hook and unhook so I used a pair of pliers to twist them off. On the left side there is a mesh cutout big enough for a 40mm fan and on the right side it’s almost entirely mesh. Weird for a rackmountable device to have side-to-side airflow but what do I know. The bottom is entirely bare and so is the top with the exception of a Cisco logo.

Guts

There are two PCBs/boards inside, the main board and an open-frame power supply. They are connected by a weird cable I’ve never seen before and have connectors I’ve never seen before. On the PSU side it’s a 6-pin and a smaller 2-pin that merge into an 8-pin on the main board side. The power plug is attached to the PSU through a 2-pin cable with a similar connector as well as a chassis ground point. One good thing is that they all look removable and fairly easy to probe with a multimeter. Overall The internals appear understandable so I’m fairly confident I can at least identify the issue.

Power

Even though I commented about the ease of probing with a multimeter, I’m sad to say that I didn’t have one previous to this experience. Since I mostly work with software, either configuring or writing it, there was never a real need to own a multimeter. I hopped on to one of the local electronic hardware supply stores’ website and found one for around 15 euros, the UNI-T UT131C. It arrived the next morning so I got to work and confirmed what repair videos from similar switches said, the main board uses 5V power. The thing is, in those other videos the voltage was lower and that was what was causing issues. In my case it was nice and steady at 4,98V which is within margin of error meaning nowhere near enough to cause a malfunction. You can never be certain though so I had ordered some alligator clips which I used for powering the main board from an ATX PSU’s molex connector. It didn’t result in a normal boot but I at least got a power light. This ruled out my main hypothesis this far which was that the switch’s power supply was going bad.

Accidental Solution

In the process of hooking the switch up to the ATX power supply, I had it open for troubleshooting. Perhaps accidentally, just to confirm that the problem was still there I reconnected the built-in power supply again. After leaving for an hour or so to do something else, I came back and saw it wasn’t boot looping.

What?

Peculiar, I thought. Sure enough, plugging in my laptop to it showed a link light and the laptop got an IP address. What the… WAIT! By hovering over it I could feel the power supply putting out some heat. No way it could be this simple, right? Could it be overheating? I turned it off, closed the case and put a 120mm fan right up against the right side to pull air out of the switch. To my surprise that was adequate to keep it cool enough even while closed. This was so embarassingly simple that I had to test again. I turned the switch on, let it get to the point where it starts constantly resetting and then put the fan next to it. After about 5 minutes it started operating completely fine, I was in awe of how simply the problem was.

Fan Replacement

Moving on to a permanent solution, I needed a fan to replace then one that was missing. Looking things up online, I realized I need a 5V 1W fan that is 40mm wide and 20mm thick. However, th only thing I found was a 0.75W fan for 8,23 euros. Good enough I guess I’ll just get two of those, one for intake and one for exhaust. Funny thing about plans, the store had only one of the 0.75W version and then one that was 0.54W. Sigh… Fine, I’ll see how I’ll arrange them. A day later they came in, I tested them both out to see that they worked and put in the 0.75W fan just to be sure the internals get enough airflow. After a 24-hour testing period, I think it’s safe to say that the problem is solved.

Use Case

The current plan is to use the switch for a VLAN so I don’t have to waste multiple ports from my main 24-port switch and be reconfiguring them. Yes, port-based VLANs aren’t ideal but I need a place to test my next network remake without bothering everything else. Even with the new fan it’s not very quiet probably due to the whine that small fans make but I have plenty of noisy things in my room.

Conclusion

It was a long and weird journey but it ended well and I now have a new switch. The troubleshooting was fun, as it always is, the new tools are fun to have and I got a lesson about assuming the problem is more complicated than it is. Thank you for reading, I hope you enjoyed the journey cause I sure did.

Flux Update

Wed, 25 Aug 2021 00:44:34 +0300

I wrote about my journey of choosing a continuous deployment tool and why I ultimately ended up using flux for my homelab in this post. It took quite a bit of reasearch and if you’re in a similar position as I was it might prove to be helpful.

Long(ish)-term experience

I’ve been using it for about 5 months now, the first post was written about one month of use. At that point I had been tinkering with it and attempting to get all the stuff I wanted to be deployed by it. Some things were simple, others not so straight-forward.

Successes

Deploying raw manifest applications

Deploying raw kubernetes manifests was really easy. Since I’m somewhat on the fence about Helm, I prefer using normal kubernetes manifests where possible. This meant that for my use case mostly everything was smooth sailing after figuring out how to generate a kustomization yaml using the command-line tool.

Command-line tool

Speaking of that, flux has a command-line tool to interact with the controllers running inside the cluster. It’s also the way you can initialize a repository to be used as the place for flux files to be stored. It can also be used to install flux to the cluster if the repository already exists.

Upgrading

When a new version of flux comes out, you can upgrade and it will carry over to the cluster. It’s really easy, it just upgrades its own files so when those are committed it will pull in the new version of the controllers.

Trouble

Aggregation yaml

For starters here is a minor one that took way too long to diagnose. I forget exactly what this is called but essentially it is a yaml file says run deal with the listed resources. Imagine the following directory contents:

drwxrwxr-x 4 angle angle 4,0K Αυγ  25 01:23 .
drwxrwxr-x 5 angle angle 4,0K Ιουλ 21 03:42 ..
drwxrwxr-x 2 angle angle 4,0K Αυγ  13 15:55 helmrepos
-rw-rw-r-- 1 angle angle  219 Ιουλ 21 03:20 infra-source.yml
-rw-rw-r-- 1 angle angle  126 Αυγ  12 11:45 kustomization.yaml
drwxrwxr-x 2 angle angle 4,0K Αυγ  13 15:56 storage

The kustomization.yaml file is as follows:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - infra-source.yml
  - helmrepos
  - storage

So it’s a way to more easily include stuff. The issue I had is that I named the file kustomization.yml instead of kustomization.yaml which for some reason isn’t supported. I believe this is a bug and not intended behavior but it was still frustrating to find and fix.

Logs

Now on to more substantial issues. While troubleshooting I got a bit frustrated with how flux displays errors. The messages are fairly non-descript and there isn’t a lot of guidance for how to fix them.

Conclusion

Despite a couple rough edges, flux has been working incredibly well. Even before I got it fully functioning, it was amazing to be able to just install it to a newly creted cluster and have all my applications running just like that. It’s almost magical and I’ve been enjoying it quite a lot. If you aren’t using a continuous deployment tool, feel free to give flux a try or read the previous post to learn about some other options

Droning On

Sat, 01 May 2021 18:54:59 +0300

I described previously how the deployment of applications is automated in my homelab but didn’t touch upon how I test my own code. The topic came up when I started trying to simulate a development pipeline who’s first step after committing code is to be run through the continuous integration system. What should that system be though? In this post, I’ll take you through the things I tried first and then why I went with drone in the end.

Choose a Fighter

My quest is to ultimately end up with the skills to set up kubernetes so that it can be deployed to everything from a DIY NAS and a stack of NUCs to a warehouse full of servers backed by distributed storage. I’m obviously more towards the first camp so that’s where my journey is at. With that said, my goal is to have software small and lightweight enough that it can run on the former setup easily so that there can be multiple instances of it in the latter form. The current mental model I have is developer organization with not much more than 3 NUCs + 1 NAS per team without losing the ability for continuous integration and continuous deployment. Therefore the landscape needs to be explored and the obvious titan of the old world is jenkins with many newcomers eager to take its place. The obvious caveats still apply, it must be open-source, able to be self-hosted, support for gitea is needed and being able to run well on kubernetes is a foregone conclusion.

Jenkins

It should need no introduction, most developers are familiar with the trusty crusty old butler software. Jenkins is probably one of the most prominent pieces of CI/CD software that was pushed to the forefront when the push for more automation and increased code quality really happened. Sadly there are a few reasons that annoyed me from the get-go. First, the unlocking procedure requires you to kubectl logs <podname> in order to find the unlock key which I understand but nonetheless find annoying. Second, the Jenkinsfile has its own domain-specific language (DSL for short) that I find unappealing even though you won’t catch me praising YAML’s lack of curly braces. I begrudgingly went though it and wrote a Jenkinsfile to give it a fair chance but it wasn’t a nice experience. Next, a single instance uses too much ram to the point that it can barely fit in my k3s VMs which use 2.5GB of ram each together with other software. This is an issue because I like running at least 2 instances of most applications to be sure that everything works correctly when scaling up. Last but not least, it’s a 2-in-1 solution which I don’t find appealing as far as architecture and since I was planning to find a standalone piece of software to do CI and another one to handle CD, it was a hard sell from the start. As a redeeming quality it has an official kubernetes pipeline plugin which is nice so they’ve at least put thought into this use case. Ultimately it wasn’t the right fit for me so it was time to move on.

Drone

While pondering this I remembered an old post from one of the blogs I read, specifically this one. I said what the hell, why not and started looking its website. Drone is a very nice and simple continuous integration tool. From the start it seemed like it would be a good pick but you never know until you try. I wrote some kubernetes manifests, got angry until I got them to work and a half hour later it was hooked up to my local gitea instance and ready to use. I added a .drone.yml to one of my repos, pushed the change, activated the repo from the drone dashboard, made another change to the source code, pushed it and waited to see what happened. Instantly drone received the webhook event and got to work. The dashboard is pretty simple (and sadly lacks a dark mode) so you just hit the thing you want to look at, click on the build and see it happen. It supports different things running at the same time like databases if you want your tests to include that which can be run either as what drone calls services aka run in the background from the beginning or if you want an ordered dependency you can use detached steps which is especially useful in microservices. My only real complaint aside from the lack of dark mode is the inability to trigger a build manually without pushing but that’s only an issue when you first activate a repository inside drone’s dashboard. All in all, it has been a joy to use, most of the projects I am currently developing have a .drone.yml and get built at least 3 times a day when gitea mirrors the gitlab or github repo they’re mainly hosted at.

Conclusion

This wasn’t as big of a journey as finding a continuous development tool because really I only found the two aforementioned options that had the basics I was looking for and one of them didn’t really fit the bill. I’m not huge into software development, it’s mostly a hobby for me as might be obvious by now, since my main concern is infrastructure-related stuff. With that said it’s useful to have the skills to build an application that can communicate with the kubernets api to automate something or write an ansible module and most of all it allows me to see the developer perspective. I haven’t started looking into SOPS or renovate bot or the k3s upgrade controller or velero but I hope to get the chance to do so in the future. Thank you for reading, I hope you enjoyed it and maybe learned something.

Fluxing My Cluster

Sat, 17 Apr 2021 17:24:22 +0300

It’s no secret that I’m a fan of automation and making life easier (even if I made it harder in the first place). One of the issues I’ve been having in my homelab is dealing with deploying stuff to kubernetes. Initially I wanted to just add a plugin to drone and be done with it. However, that didn’t really pan out which ended up being to my benefit since I discovered flux.

The problem

The cluster in its current state is a little sad because I end up doing too many things manually. Whenever something changes it’s time to bring out ye old kubectl apply -f in a local clone of the repo and that doesn’t spark joy. This was obviously bad form and couldn’t continue. What you also have to keep in mind is that I’ve chosen a 3rd-party storage plugin, democratic-csi and assume it’s the default storageclass in the cluster. This means that on a new cluster it’s required that before anything else, that one is added as a storageclass and set as default and local-path is removed from being a default. After that, the applications can be deployed which includes a mix of stuff I’ve written for the purposes of testing as well as 3rd-party software. Now that the stage has been set, let’s see what I tried

Solutions

Drone CI plugin

Initially the idea was that since I’ve alrady solved this for code using drone for CI (this will be discussed in a later post), I could just use a plugin and be done. However, my experience with writing helm charts is limited and the only official plugin for deploying to kubernetes is for helm charts, not kubernetes manifests. Furthermore, as I mentioned previously, not all software deployed on the cluster is something that I have written so even if there was an official kubernetes manifest plugin it wouldn’t cover 3rd-party software. At this point, I was thinking of finding some all-inclusive CI/CD software that would cover everything I wanted to do. It took no more than 10 seconds of thinking to realize that I was looking for non-modular software and discard that idea.

Standalone Continuous Delivery

This was the obvious choice but the software in this category is plentiful. The main competitors were flux, argo and tekton. Their designs differ quite a bit so there is going to be quite a bit of opinion in the following analysis so just keep that in mind. The few things they had in common were that they were all advertised as cloud-native with support for kubernetes, had a cli and were written in Go.

ArgoCD

I started with argo because one of the people that wrote a really good blog post on using kubernetes at home mentioned that they use it so I thought I’d try it first. The installation process required a massive yaml file that you just curl | kubectl apply -f which I wasn’t a big fan of but alright whatever. It also had a weird unlock procedure like jenkins where you have to find a token generated at runtime to unlock it but that’s okay, just an one-time setup thing. After spending a couple evenings with it, I wasn’t satisfied with the experience. The docs weren’t really good at explaining all the argo custom resources and getting a basic single-pod application running took me more than 2 hours. It was doable but this was going to be something I had to do for every application running on my cluster and the prospect didn’t seem appealing. I’m sure with more time I could maybe become more familiar and eventually warm up to it however first impressions were bad so I decided to move on.

Tekton

Tekton’s marketing was interesting so I took a look at it next. It seemed to be easier to set up and a couple random blog posts seemed to praise it so I downloaded another massive installation yaml which the guide told me to just kubectl apply -f which is okay I guess. I spent a few hours with it and it became clear you basically had to use it both for CI and CD so I put it to the side and moved on.

Flux

So why did I mention this one first but try it last? A few reasons. While researching, the v1 -> v2 development effort was going on and v2 didn’t have a lot of features but v1 was being phased out so I didn’t know if I should spend time learning v1 while v2 was coming along or if it’s worth jumping into alpha/beta stage software to avoid using legacy versions. By the time I did get around to it though v2 was clearly the way forward. Not only was it officially endorsed but among people that self-host, it was already the way to go 6-12 months ago. The docs were pretty clear, the cli was cross-platform and could be installed on a Raspberry Pi (something that argo at the time didn’t have available) and the examples worked. In an evening I had learned basically everything I needed to in order to just drop it in, generate the flux CRDs for my existing kubernetes manifests as well as helm charts and be up and running. The only issue I had, probably just a pet peeve of mine, was that the repo it creates on github is private by default and my goal is to have everything public and also secure (not there yet) but whatever. Flux v2 also mentioned as the gitops toolkit is made up of several different parts that have a specific purpose which is something I certainly appreciate. All in all, I think I’m going to stick with flux but you never know how homelab things will end up.

Conclusion

Flux is a really great project and I’ve enjoyed using it so far. The problem of continuous delivery has been solved for me and I recommend checking out if you’re in a similar position. Next up, I’m going to look at renovate bot for automating image updates, SOPS for secret management (which flux supports and an upgrade controller) to further help automate operations. Thank you for reading, I hope you enjoyed it and maybe learned something.

Homelab Current Form

Sun, 07 Feb 2021 14:35:32 +0200

The first part of the software changes I made was covered in the first part where I explained how and why I started going down this path.

Software choices

This whole journey was about making my home infrastructure better. Part of that was about having a way to more easily create the foundation on which kubernetes would run on, as well as describing my setup as code. Proxmox is a beast in its own right. Features include being able to manage kvm virtual machines and lxc containers, an API that can be used to interact with it programmatically and most importantly being open source. While researching what automation tools could interact with proxmox, I found a community-made terraform provider as well as a set of two ansible community modules (kvm and lxc). After thinking about it a little bit, I wanted to use terraform in a non-cloud environment but also gather some knowledge about it so that’s what I decided to use. I can have my provisioning requirements in a file that terraform understands and store it in git which is exactly what I was going for. Ansible is still in the picture since it is used to set up and configure the debian environment inside the virtual machines. Now that the hypervisor part is covered, let’s move on to running services.

Making it store is harder than making it run

As mentioned in the first part about software changes, I was learning kubernetes and now I could not only wipe and recreate the cluster itself but also the entire virtual machines that it was running on. Some time after starting to use proxmox with terraform I attempted to use rancher to manage kubernetes but ended up ditching it due to various problems with running even basic stuff on it (very likely that it’s a case of PEBCAK, I don’t think rancher is terrible or anything like that).

The problem

However the problem of shared storage continued to taunt me for the following months. There was seemingly nothing that a simple fella could do to use a nice and simple NAS running truenas core as storage that is able to be dynamically provisioned for use by kubernetes. Now, I hear you, “what about nfs-client-provisioner”, someone less familiar with this pile of madness might exclaim. Indeed it does exist and barely work except the helm chart for it is deprecated and it does not work with kubernetes version 1.20 and later since it does not seem to be using CSI drivers.

I really tried

Months of furious and frustrating testing ensued. Not only was I trying to get applications running on kubernetes, I was also fighting with the unexpectedly complex tast of using the storage server I had available.

glusterfs

I’ve gone through basically any and all commonly suggested options for dynamically provisioned storage. Prette early on I tried glusterfs with heketi by making 3 LXC containers and mounting an nfs share to each one that would serve as the brick in the gluster volume. Suffice it to say that it didn’t work and things were getting out of hand.

longhorn

After a bit more research and I found out that longhorn, another project by the authors of k3s, used iscsi to communicate with kubernetes. That could work I thought, except there was no coherent example of how to use it without using targetd for iscsi. Longhorn was starting to look more appealing, I could just put the virtual drives of the VMs on an nfs share, run longhorn inside the VMs to pool all their storage together and call it good enough. No, I could not give up yet. A sub-optimal solution would do if there was no other way but I was convinced something more was out there. My patience and hope were running out but not empty yet.

success

That all changed in early January where during my nearly daily search for possible storage solutions I hadn’t tried, I found out about democratic-csi. This was it. Made to be used with freenas, truenas as well as DIY ZFS setups. The silver bullet was here. Just a few minutes of reading about it and writing my configs, a short(read: long) helm command later and…success! The test pod was using the newly created storageclass that was backed by the nfs share on truenas. After that I rushed to deploy my standard set of gitea, droneci and minecraft to test it out and it was working for real without a hitch.

Conclusion

This was a long journey and it isn’t coming to a close any time soon. The next step is to continuous deployment but that’s an issue for another time. To conclude as briefly as possible, I’ve no put most of the critical pieces of my homelab in a single git repo that I can use to recreate almost all of it from scratch with minimal manual intervention (there are still a few quirks like not having dynamic inventory for ansible and having to manually copy IPs but I can put that aside for now). Despite the imperfections like not having dynamic inventory for ansible or having secrets stored unencrypted in git, I’m very happy with the setup is working. If you missed them, make sure to check out the hardware updates and the first part about software changes for a better of what I’ve been up to.

Homelab Evolved

Sun, 07 Feb 2021 13:53:10 +0200

The hardware side of the changes made to my homelab were covered in a previous post where I also alluded to some software changes.

What came before

This is the first part explaining more in-depth the issues that I had and how I dealt with them. Let’s set the stage first, my homelab was working fine in many ways. It was nice and stable debian install running kvm virtual machines using qemu and libvirt with virt-manager to perform the simpler start/stop tasks. There was really nothing wrong with it, I was exploring different software, running some services on the local network and even hosting a public website on it.

Good enough is not good enough

However as someone interested in infrastructure a few things were bothering me. Sure, I was using docker-compose with traefik for my public website and services. Yes, I was using ansible to manage the configuration of most virtual machines. But “good enough” and “I guess it works” doesn’t cut it. On the upside, I really liked having ZFS snapshots on a 2x2TB mirror. The ability to go back to a point in time where things were not broken had saved me a couple times when I inevitably broke one thing or deleted a file that could be replaced but would take a while. My backups were also on that ZFS mirror, on a different dataset of course, and that also very useful. I knew I wanted ZFS and that there was much room for improvement in regards to handling how I was running services. The single docker-compose virtual machine for the public stuff and another one for local stuff were mostly adequate but the manual management was not. Not to mention that and all my precious stuff was on one box which is well below ideal.

Taking the step

One day I decided it was time to move up in the world. Despite using docker and docker-compose for years I had never dived into kubernetes because it seemed difficult and complex. New syntax, new system with its own architecture and internal structure, new workflow, new everything. I decided to set up 3 VMs and played with default k8s as well as k3s on them. Dipping my toes into it was pretty fun and since nothing depended on it I was free to wipe it and start over. The same issue kept cropping up, having to manually install debian thrice and set up kubernetes just to delete all of it and redo it was again, less than ideal Initially I started writing a playbook before finding out that someone smarter had already gone down this path. Setting up k3s-ansible was easy so there was at least kubernetes setup automation but the whole thing was too thrown together there was no shared storage. However, I was becoming more familiar with kubernetes and learning the concepts as well as how to write yaml manifests.

Conclusion

At some point it became too bothersome, a lot of manual work was involved and basic stuff like copying over ssh keys was also done mostly manually so I had to move on. While all of this was going on, I had started planning what hardware updates I would get which you can read about here: Hardware Updates 2020

Hardware Updates 2020

Fri, 15 Jan 2021 21:10:38 +0200

Getting a real storage server

During 2020 I got a few upgrades to my setup but I don’t remember the exact order of them. The most exciting was a 4U rackmount case that I could use to put up to 6×3.5″ hard drives by default and up to 9 with a 2×5.25″ to 3×3.5″ hotswap bay. Also I bought 3x4TB ironwolf hard drives to put in it initially because my 2x2TB mirror was regularly reaching 95% capacity meaning I had to prune the snapshots regularly and couldn’t use it to store more files on it. This made me warm and fuzzy inside but the old mirror had to sit on the bench for this transition. The power supply in my server had only 4 sata connections which would be enough for a boot SSD and one of the arrays but not everything at once. Initially that wasn’t a problem, installed truenas core on the ssd, set up the 3 4TB hard drives in a raidz1 configuration and started creating datasets and shares.

Adding some extra spice

After a while I wanted to access the data on the old miror which meant having to buy a new power supply so I got one. Namely the Corsair CV550 which has 7 sata power connectors, more than enough to power the five hard drives and the one SSD. Moving on to issue number two, data connections of which the motherboard only had 4 of. Admittedly 50 euros for an extra 4 sata ports seemed excessive but I saved up and acquired a nice and simple pcie card that did the job, no raid controller of course. Alright, we’re cooking with gas now, storage is taken care of.

Dealing with separation of concerns

However, I used to have a multi-purpose server and now it is just a NAS so I can’t run all my services anymore. Fear not because the trusty old and used Dell Optiplex came to the rescue. A shop near my house had one with an i5 2400 and 4gb of ram as well as a 250gb hard drive with windows on it for just 130 euros so I jumped on it. First things first, I can’t be using a mechanical hard drive for the operating system and proxmox doesn’t dual boot so the HDD has to be replaced. Luckily the 240gb SSD from when I originally built my desktop is still alive and it’ll do more than fine for this purpose. The hard drive will be put aside for now but rest assured I do plan to use it at some point.

Serially accessing memories about ram upgrades

More importantly, I got a handle on the ram situation of my NAS. Upgraded that beast from 12 to 16 gigabytes of ram which left the 4gb stick as a spare. Not being one to miss out on some extra ram, the spare got added to the optiplex. Going from 4 to 8 is nice but I want to run more than 2 virtual machines. With that in mind and a deal available, I added an 8gb to the optiplex. So we’re up to 16(8+8) gigabytes for the NAS and 16(2+2+4+8) gigabytes for the server.. An odd configuration but ram is ram and you bet I’ll use it.

Conclusion

Perfect, we had storage and now we have compute as well. How was this hardware used though? Check back later (or read the next post if it’s up) to find out.

Homelab on inherent site

Home Assistant

Getting started

Graphing the worst star’s waste heat

Add-ons

Later expansion

Conclusion

Remote Access Homelab

Tailscale

Cloudflare Tunnel

Conclusion

Talos Linux Setup and Configuration

How the Talos configuration works

Separate secrets

Patches

Conclusion

PXE setup for Talos Linux with Matchbox

Plan

PXE

Talos

Matchbox

Definition of Done

Implementation

How do I do X in Y minutes

Explanation

Conclusion

Homelab 2025 Plans

Intro

Hardware

Software

Kubernetes

Router

Conclusion

State of Homelab 2024

Router

Virtualization

Kubernetes

NAS

Hardware

Conclusion

QEMU Guest Agent for Terraform on ProxMox

The Problem

The Solutions

Images with qemu-guest-agent pre-installed

Terraform remote-exec provisioner

Packer proxmox-iso Builder

Adding qemu-guest-agent using guestfs-tools

Conclusion

Network Design Document

Network Layout

VLANs

Conclusion

Ansible, Terraform and Proxmox

Proxmox Setup with Ansible

Host

Repository

Template VM

VM Image

Template Script

Terraform Setup

Provider

Basic main.tf

Creating a test VM

Creating VMs for k3s

Conclusion

Cisco SR2024C Repair

Backstory

Initial Findings

Open Wide

Guts

Power

Accidental Solution

Fan Replacement

Use Case

Conclusion

Flux Update

Long(ish)-term experience

Successes

Deploying raw manifest applications

Command-line tool

Basic `main.tf`