Networking on inherent site

PXE setup for Talos Linux with Matchbox

Sat, 01 Mar 2025 19:00:33 +0200

Join my campaign of hate against installing operating systems interactively. I usually wipe my homelab virtualization hosts every week or two. This might sound insane but I’ve been working on automating their setup. As explained in the previous post, friendship ended with ProxMox and kubernetes will take its place. Specifically Talos Linux, a linux distro you can’t even SSH into. ProxMox kind of notoriously doesn’t have a way to be sanely installed through PXE but Talos is kind of made for it. However, PXE (and iPXE, we’ll discuss this later) is not perfectly documented and can be even harder to debug. So in this post I’ll document what worked for me and talk about some pretty helpful resources.

Plan

PXE

So let’s start with a bit of background, then lay out what I wanted to achieve, why and what I considered “good enough”. First off, PXE. It stands for pre-boot execution environment and is mostly reliant on motherboards and network cards to be implemented on the client side. It’s a stripped down environment before the computer is booted into an operating system. Inside it, there are standards for getting information about what to execute, where to get it and what configuration to use. PXE is the name of the general concept and the first version of the technology. It allows booting a computer from the network by loading files from a TFTP server. Later on, iPXE came around to fix some of its limitations. One thing it’s useful for having a computer and being able to boot it from the network and install an operating system on it without having to have a USB stick on you. With something like syslinux you can even create menus to be able to choose what to install and have a bunch of different options with different settings. As computer installations grow larger, it’s easy to see how this becomes kind of essential as you can’t run around multiple datacenters any time something needs to be re-imaged. A counter-point to that could be that servers have remote management features allowing us to load ISOs as if they’re DVDs but even then, iPXE has an advantage because you can fully automate the process. Also my poor version of servers (read: used office PCs) do not have IPMI. At any rate, that’s the general idea of PXE.

Talos

How does that apply to my case then? I want to be able to completely wipe all three systems and then effortlessly restore them. As I plan to run kubernetes on it, the installation of applications to the cluster is taken care of using FluxCD so I only really need to solve the installation part. That’s where Talos comes in. Talos is an incredibly minimal linux distribution that is designed with the sole purpose of running kubernetes. No SSH, no TTY, no nothing. Sitting in front of the system only allows you to change kernel boot parameters and the network configuration. Everyhing is done using the API which is accessible interactively through the talosctl CLI utility. This means that we can have our operating system and kubernetes configuration as code for the price of one config file. Sign me up!

Matchbox

Those are the ingredients but it can hardly be called a meal. We need to get some kitchen utensils in this analogy to get the desired result. Our PXE setup will need to have a little bit of logic in it so it knows what to install with what configuration. See, Talos can be told to fetch a configuration file from an HTTP server using a kernel parameter. A conditionally-rendered file would allow us to alter the response depending on who’s requesting it. This is where matchbox slots in. When matchbox gets a request it can see some information about the host requesting it and therefore make a determination about what to reply with. We can then serve different options according to which system is asking such as what Talos configuration file to use. This way the control-plane kubernetes nodes can receive the Talos controlplane.yaml while worker nodes get the worker.yaml. So a brand-new host with an empty disk drives would boot up, skip the disks since they don’t have an operating system and then try booting using PXE. Then the host would get its configuration, install the operating system, install kubernetes and become part of a cluster.

Definition of Done

The ideal process when getting adding a host should look like:

note down the MAC address and corresponding IP somewhere
plug in the cables
configure the boot device order
save settings and exit
play minecraft while waiting
run one or zero commands
host shows up in kubectl get nodes
recreating it from scratch should be just as easy

I don’t know if I can get exactly to that point but I’m willing to put in some effort into the setup so in the future I can avoid it to a greater extent. Some compromise is acceptable since I don’t have any IPAM infrastructure in place already so this isn’t the full and proper way to go about it.

Implementation

How do I do X in Y minutes

Meat and potatoes first, for quick reference here is what I did (5-6 hours of suffering omitted for brevity):

Installed tftp-hpa on Arch
Downloaded undionly.kpxe and ipxe.efi
Put them in /srv/tftp

Created matchbox.ipxe with the following contents:

1
2

#!ipxe
chain http://10.0.20.50:8080/boot.ipxe

Installed matchbox using Docker by running the following command:

`1`	`docker run --net=host --rm -v /var/lib/matchbox:/var/lib/matchbox:Z -v /etc/matchbox:/etc/matchbox:Z,ro quay.io/poseidon/matchbox:v0.10.0 -address=0.0.0.0:8080 -log-level=debug`

Downloaded vmlinuz-amd64 and initramfs-amd64.xz
Put them in /var/lib/matchbox/assets
Download talosctl
Run talosctl gen config homecluster https://10.0.50.69
Edit controlplane.yaml until it looks good
Put controlplane.yaml in /var/lib/matchbox/assets as well

Create /var/lib/matchbox/profiles/control-plane.json:

{
  "id": "control-plane",
  "name": "control-plane",
  "boot": {
    "kernel": "/assets/vmlinuz",
    "initrd": ["/assets/initramfs.xz"],
    "args": [
      "initrd=initramfs.xz",
      "init_on_alloc=1",
      "slab_nomerge",
      "pti=on",
      "console=tty0",
      "printk.devkmsg=on",
      "talos.platform=metal",
      "talos.config=http://10.0.20.50:8080/assets/controlplane.yaml"
    ]
  }
}

Create /var/lib/matchbox/groups/control-plane.json:

{
  "id": "kube01",
  "name": "kube01",
  "profile": "control-plane",
  "selector": {
    "mac": "18:03:73:2a:d8:a2"
  }
}

In OPNSense (link to my router)
- Under DHCP Static Mappings for this interface. add the hosts
- Under TFTP server:
  - Set TFTP hostname: 10.0.20.50
  - Set Bootfile: undionly.kpxe
- Under Network booting
  - Set next-server IP: 10.0.20.50
  - Set default bios filename: undionly.kpxe
  - Set iPXE boot filename: matchbox.ipxe
Boot Optiplex
Wait
Watch text scroll on screen
???
No profit, I spent my whole weekend on this

Explanation

PXE is an under-documented part of the common network services one can set up. An amusing yet educational thing I found was a redditor’s 3-part descent into madness and consequently enlightenment:

The matchbox documentation was also very good at explaining both the general process and giving some specific recommendations:

With those out of the way I’ll make an attemt at explaining it myself. When a PXE client starts, it does the DHCP dance which tells it where the TFTP server is and what the name of the file it should execute is. That file is a network boot program which can optionally load some configuration (also stored on the TFTP server). So after all of that is figured out, the PXE client will pull the kernel and initramfs, again from the TFTP server. In contrast, iPXE uses scripts stored on the TFTP server instead of plain configuration files so we can do fancier things with it. However, my old desktop PCs do not have iPXE so I had to use an in-between thingy, undionly.kpxe. What that does is load iPXE firmware (ipxe.efi) so we can access the cool new features without having to replace any hardware. When that loads, it talks to the DHCP server as an iPXE client this time and gets told to grab the matchbox.ipxe file from the TFTP server. That file is an iPXE script which tells the client to check the matchbox server. When matchbox gets the request, it checks the groups it has registered and if there are any matches according to the selector field, it uses the corresponding profile. In this case, the profile is for control-plane Talos Linux nodes. The kernel and initramfs listed in the profile are used alongside some specific kernel parameters. The most important parameter is talos.config=http://10.0.20.50:8080/assets/controlplane.yaml which tells the machine that is booting the operating system to fetch the configuration file listed there. Of course, you will need to have generated the configuration (using talosctl gen config homecluster https://10.0.50.69:6443) and moved it in the matchbox assets directory (or in another HTTP server, matchbox is just convenient in this setup). That configuration has some basic things like which disk and network settings to use. When the machine gets this configuration, it installs Talos and then waits. At this point the system is booted normally, no PXE involved.

Following that, all we need to do is run talosctl bootstrap --nodes 10.0.50.71 --endpoints 10.0.50.71 and we get a kubernetes cluster. To get the kubeconfig required to access the cluster, run talosctl kubeconfig my-talos-kubeconfig --nodes 10.0.50.71 --endpoints 10.0.50.71 and then check that it works by running kubectl --kubeconfig my-talos-kubeconfig get nodes. I’ve tested the whole process end-to-end twice now to ensure that nodes with blank disks and correct boot order will behave correctly and it’s almost magic. True enough, after pressing the power button on all three systems, making some coffee, and then running the commands mentioned previously, I got this output:

$ kubectl get nodes -o wide
NAME     STATUS   ROLES           AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION   CONTAINER-RUNTIME
kube01   Ready    control-plane   3h20m   v1.32.2   10.0.50.71    <none>        Talos (v1.9.4)   6.12.13-talos    containerd://2.0.2
kube02   Ready    control-plane   3h20m   v1.32.2   10.0.50.72    <none>        Talos (v1.9.4)   6.12.13-talos    containerd://2.0.2
kube03   Ready    control-plane   3h20m   v1.32.2   10.0.50.73    <none>        Talos (v1.9.4)   6.12.13-talos    containerd://2.0.2

And I guess that’s it. It needed some effort and research and the approach should be adjusted according to the network situation (the matchbox docs saved the day here). In the end though, I really like what I got working and it was a fun way to spend the weekend. I can go from basically brand new computer with nothing on it to full cluster with a high availability control-plane using patience and a single command. Two commands and I can install fluxcd on the cluster which will install all my applications. Looking back at the definition of done I set in the beginning, I consider this a success.

Conclusion

As the proud owner of a Talos Linux kubernetes cluster installed in a fully automated way, I am quite happy. I can get back into running services at home and live through the pain of using kubernetes daily again. I have a couple of ideas in mind for how to use my new container running platform but that’s for another post. Hopefully you got a better idea of what PXE is, how it works and how I used it in this instance. If you weren’t familiar with any of this, maybe it’s time to get your hands dirty and prepare to throw away your USB drives. Either way, that’s all for now.

Virtualized pfSense on ProxMox

Tue, 20 Sep 2022 01:12:49 +0300

Recently I went down the path of running my router inside a virtual machine. This is far from the first time I’ve tried it but it is the first time I have a solid long-term plan. Keep reading if you’re interested in doing something similar, this should be fun.

Background

Last time I ran my router inside of a virtual machine was in 2017 and it was documented in this article. The reason, at least as far as I can tell, that it didn’t stick is because I ran it on my desktop. Since then, I started my homelab where I can run software on machines that are online most of the time. A small Dell Optiplex is more than enough for my networking needs at this time and I have two of them. I picked the weaker one since it isn’t as useful for quickly spinning up and down VMs due to the old CPU. My plan is to run some LXC containers and KVM virtual machines alongside pfSense which is why I’m virtualizing it.

Preparation

The ProxMox installation on the weak Optiplex is configured the same as the main one for testing purposes. When using the system as a staging environment, I installed it using ZFS. This was a great choice for having a 1-to-1 copy of the production version but it uses a little bit more RAM than I’m willing to spare. Due to unfortunate circumstances it’s equipped with only 20GB of memory and using more than 2gb of that for the host limits what other things can run at the same time on it. With that in mind I chose to reinstall using the default LVM scheme. After the installation I followed the official PCI passthrough guide and rebooted.

PfSense VM Setup

Since I have 2 identical Realtek NICs but different revisions, I looked through the output of lspci -vvvv to identify the correct one. With the device ID noted, I created a q35 UEFI-enabled virtual machine and made sure to uncheck the Pre-Enroll keys option (that one took a while to figure out). Don’t forget to set the option Start at boot to make it operate as if it was a bare-metal installation. The installation process is very streamlined so I was up and running very quickly. I had to do a little dance of plugging/unplugging cables to my switch and assign a spare VLAN to a port for setup purposes but once that was done I uploaded the backup config. Reassigning interfaces and VLANs was a bit tedious however the whole thing didn’t take more than 20-30 minutes.

Results

The performance isn’t any worse and reliability is somehow increased despite the old box using an Intel NIC and the new one using Realtek. As far as consolidation goes, this is a huge increase and I can now run multiple other things on a single machine without relying on pfSense packages.

Network Design Document

Sat, 16 Jul 2022 18:45:44 +0300

I recently started remaking my network at home and decided to publish some of the design documentation here

Network Layout

VDSL Internet comes in through an RJ11 port to a Vodafone H300s modem-router. LAN port 2 of the H300s is plugged into port 2 (untagged VLAN 10) of the Netgear GS724Tv4 switch. Switch port 23 (untagged VLAN 1, tagged in all other VLANs) is plugged into the gigabit ethernet port of the pfsense router. All devices are plugged directly into the switch since they fit. Inter-VLAN routing happens through pfsense and the switch operates as a managed L2 switch even though it has L3 capabilities.

VLANs

Probably overkill but certainly enough:

VLAN	Description	Subnet	Gateway	DHCP range from pfsense
1	Default	-		-
2	Auto VoIP	-	-	-
3	Auto-Video	-	-	-
10	Border Router LAN	10.0.10.0/24	10.0.10.1	-
20	End Devices	10.0.20.0/24	10.0.20.254	10.0.20.10-245
22	Trusted	10.0.20.0/24	10.0.22.254	10.0.22.22-222
30	Monitoring	10.0.30.0/24	10.0.30.254	-
40	Storage	10.0.40.0/24	10.0.40.254	-
50	Servers	10.0.50.0/24	10.0.50.254	-
60	Guest	10.0.60.0/24	10.0.60.254	10.0.60.2-252
65	Media	10.0.65.0/24	10.0.65.254	10.0.65.2-252
66	IoT	10.0.66.0/24	10.0.66.254	10.0.66.2-252
67	DIY IoT	10.0.67.0/24	10.0.67.254	10.0.67.2-252
70	Security	10.0.70.0/24	10.0.70.254	-
80	DMZ	10.0.80.0/24	10.0.80.254	-
90	APIs	10.0.90.0/24	10.0.90.254	-
99	Management	10.99.0.0/16	10.99.0.1	-
111	Kubernetes	10.0.111.0/24	10.0.111.254	-
113	MAAS	10.0.113.0/24	10.0.113.254	-
123	Windows	10.0.123.0/24	10.0.123.254	-
150	Test LAN	-	-	-
666	Native	-	-	-
1022	Unused Port	-	-	-
1337	Danger	-	-	-

Conclusion

That’s my home LAN currently, I’ll try to keep this updated as I change stuff but no promises.

Cisco SR2024C Repair

Mon, 28 Mar 2022 18:03:46 +0300

One of my friends at university gave me a broken switch that seemed broken and I decided to fix it. Join me on a pretty entertaining troubleshooting path that was not entirely necessary.

Backstory

This semester in university I am taking a CCNA class. The learning path is split across two semesters, next semester I’ll take CCNA 2 to be ready to get the certification. A few of my friends are also there an one of them happened to have a supposedly broken Cisco switch. Great opportunity to use it for hands-on learning, I thought. He said it wasn’t fully broken, it could turn on fine but would turn off after 10-20 minutes. Not that bad. A little fan was also missing from it but that would be no more than 5-10 euros to replace. He gave it to me one day, I took it home and that was that. After a bit of research I realized the switch wasn’t managed meaning it didn’t have Cisco iOS capabilities. Nevertheless, I like tech necromancy and network gear seldom goes unused. So that’s basically how I got a Cisco SR2024C.

Initial Findings

After checking that the power plug looked fine and the ports didn’t look damaged, I decided to turn it on. Indeed, after roughly 20 minutes it started boot-looping so the behavior was as described. The switch wouldn’t completely shut off, instead it followed a little pattern:

lights flash for a second the way they do when first plugged in
lights go out same as when normally booting
goto step 1

This meant that eventually anything plugged into it would drop off the network which is not good. At this point I’m suspecting the power supply since the device is pretty old, about 7 years past its End-Of-Life date.

Open Wide

Behavior confirmed and reproducible, we have a place to start from. The device would need to be opened anyway in order to replace the fan and I’d like to inspect the inside too. This specific switch is considered compact as noted by the ending C in its model number (SR2024C). Its width is a bit more than the width of the ports, 24 RJ45 gigabit ethernet ports and two miniGBIC. It has holes to attach rackmount ears if desired but I wasn’t provided those. On the front there is a plastic cover that fits over the 4 sets of ports (3 sets of 8xRJ45 and one set of the 2 miniGBIC) and also clips on to the top and bottom metal pieces. I dinged the plastic since I used a pair of scissors to pry it off but that’s alright, we’re going for functional not pretty. Following that, I had to carefully unhook some little metal tabs that held the top half of the metal case to the bottom half. Push a little here, pull a little there and ta-da! We’re in. The metal tabs were annoying to hook and unhook so I used a pair of pliers to twist them off. On the left side there is a mesh cutout big enough for a 40mm fan and on the right side it’s almost entirely mesh. Weird for a rackmountable device to have side-to-side airflow but what do I know. The bottom is entirely bare and so is the top with the exception of a Cisco logo.

Guts

There are two PCBs/boards inside, the main board and an open-frame power supply. They are connected by a weird cable I’ve never seen before and have connectors I’ve never seen before. On the PSU side it’s a 6-pin and a smaller 2-pin that merge into an 8-pin on the main board side. The power plug is attached to the PSU through a 2-pin cable with a similar connector as well as a chassis ground point. One good thing is that they all look removable and fairly easy to probe with a multimeter. Overall The internals appear understandable so I’m fairly confident I can at least identify the issue.

Power

Even though I commented about the ease of probing with a multimeter, I’m sad to say that I didn’t have one previous to this experience. Since I mostly work with software, either configuring or writing it, there was never a real need to own a multimeter. I hopped on to one of the local electronic hardware supply stores’ website and found one for around 15 euros, the UNI-T UT131C. It arrived the next morning so I got to work and confirmed what repair videos from similar switches said, the main board uses 5V power. The thing is, in those other videos the voltage was lower and that was what was causing issues. In my case it was nice and steady at 4,98V which is within margin of error meaning nowhere near enough to cause a malfunction. You can never be certain though so I had ordered some alligator clips which I used for powering the main board from an ATX PSU’s molex connector. It didn’t result in a normal boot but I at least got a power light. This ruled out my main hypothesis this far which was that the switch’s power supply was going bad.

Accidental Solution

In the process of hooking the switch up to the ATX power supply, I had it open for troubleshooting. Perhaps accidentally, just to confirm that the problem was still there I reconnected the built-in power supply again. After leaving for an hour or so to do something else, I came back and saw it wasn’t boot looping.

What?

Peculiar, I thought. Sure enough, plugging in my laptop to it showed a link light and the laptop got an IP address. What the… WAIT! By hovering over it I could feel the power supply putting out some heat. No way it could be this simple, right? Could it be overheating? I turned it off, closed the case and put a 120mm fan right up against the right side to pull air out of the switch. To my surprise that was adequate to keep it cool enough even while closed. This was so embarassingly simple that I had to test again. I turned the switch on, let it get to the point where it starts constantly resetting and then put the fan next to it. After about 5 minutes it started operating completely fine, I was in awe of how simply the problem was.

Fan Replacement

Moving on to a permanent solution, I needed a fan to replace then one that was missing. Looking things up online, I realized I need a 5V 1W fan that is 40mm wide and 20mm thick. However, th only thing I found was a 0.75W fan for 8,23 euros. Good enough I guess I’ll just get two of those, one for intake and one for exhaust. Funny thing about plans, the store had only one of the 0.75W version and then one that was 0.54W. Sigh… Fine, I’ll see how I’ll arrange them. A day later they came in, I tested them both out to see that they worked and put in the 0.75W fan just to be sure the internals get enough airflow. After a 24-hour testing period, I think it’s safe to say that the problem is solved.

Use Case

The current plan is to use the switch for a VLAN so I don’t have to waste multiple ports from my main 24-port switch and be reconfiguring them. Yes, port-based VLANs aren’t ideal but I need a place to test my next network remake without bothering everything else. Even with the new fan it’s not very quiet probably due to the whine that small fans make but I have plenty of noisy things in my room.

Conclusion

It was a long and weird journey but it ended well and I now have a new switch. The troubleshooting was fun, as it always is, the new tools are fun to have and I got a lesson about assuming the problem is more complicated than it is. Thank you for reading, I hope you enjoyed the journey cause I sure did.