Zfs on inherent site

Reinstalling Gentoo

Wed, 10 Mar 2021 00:54:04 +0200

Why

This gentoo install has lasted a good long couple of years. There is nothing really wrong with it, the SSD it’s installed on is a bit full and it’s a good opportunity to make a change. However I’m looking to shake things up a bit. The big change is installing on ZFS this time. It’s fairly well-known that I’m a big fan of ZFS, my storage server has been using it in one form or another sinc forever and all of my home infrastructure directly or indirectly depends on it. However it is not the only change being made, executing the install using ansible will be a good improvement too. One of the issues with gentoo, to people that have been using it for a while, is the seeming non-repeatability of the install. A script can get really close however certain steps are not as easy to get right and everything is highly procedural instead of declarative or idempotent way. The other changes will include moving to a system with as few, as I view them, harmful or unnecessary componenets. Stuff like dbus, eudev, elogind, pulseaudio, systray and a few other things that I’m in the process of nailing down will be (hopefully) gone.

How

Ansible

First off, the ansible playbook. There are a few different ones floating around but are outdated or don’t fit my needs. Of course that doesn’t mean I can’t examine the pre-existing solutions, I will absolutely take inspiration from solutions that smarter people have come up with.

ZFS

The most important decision will be the layout. I know I’ll be using docker and libvirt so adjustments to /var/lib are needed. Also, the system needs to be bootable and usable to some extent even if the ZFS kernel module doesn’t get loaded. There are 2 different solutions to this, either go with EXT4/XFS or just build ZFS into the kernel. I’m still undecided about the route I’ll go with but remain open to experimenting and seeing which one is better.

Cruft removal

Several components are impacted here. The most interesting one I’m looking forward to installing is libudev-zero to seemingly satisfy any dependencies that require (e)udev while in reality using busybox mdev. This gets rid of some dependency problems with libinput/evdev and anything else that has a hard dependency on a single device manager implementation without source code patches to those projects. It also give me the ability to use input classes in xorg instead of hardcoding device paths which is something that was somewhat annoying last time I tried switching away from eudev. As for the rest, I’m testing out what can be effortlessly removed and what requires significant changes to one thing or another on my laptop. We’ll see what I end up with, I’ll update this post after it’s up and running.

Conclusion

I’ve enjoyed my gentoo system for multiple years at this point but I want to make things better so I’ll be reinstalling and starting from a clean slate. Switching the filesystem to the one my storage server relies on will be pretty sweet (I might also look into some pre or post package install hooks). The installation process, and hopefully the full system setup, will be described using ansible to make it more easily repeatable And finally, any piece of software I don’t like or use will be removed or replaced and incompatibilites fixed along the way. I look forward to a better gentoo system in 2021.

Homelab Current Form

Sun, 07 Feb 2021 14:35:32 +0200

The first part of the software changes I made was covered in the first part where I explained how and why I started going down this path.

Software choices

This whole journey was about making my home infrastructure better. Part of that was about having a way to more easily create the foundation on which kubernetes would run on, as well as describing my setup as code. Proxmox is a beast in its own right. Features include being able to manage kvm virtual machines and lxc containers, an API that can be used to interact with it programmatically and most importantly being open source. While researching what automation tools could interact with proxmox, I found a community-made terraform provider as well as a set of two ansible community modules (kvm and lxc). After thinking about it a little bit, I wanted to use terraform in a non-cloud environment but also gather some knowledge about it so that’s what I decided to use. I can have my provisioning requirements in a file that terraform understands and store it in git which is exactly what I was going for. Ansible is still in the picture since it is used to set up and configure the debian environment inside the virtual machines. Now that the hypervisor part is covered, let’s move on to running services.

Making it store is harder than making it run

As mentioned in the first part about software changes, I was learning kubernetes and now I could not only wipe and recreate the cluster itself but also the entire virtual machines that it was running on. Some time after starting to use proxmox with terraform I attempted to use rancher to manage kubernetes but ended up ditching it due to various problems with running even basic stuff on it (very likely that it’s a case of PEBCAK, I don’t think rancher is terrible or anything like that).

The problem

However the problem of shared storage continued to taunt me for the following months. There was seemingly nothing that a simple fella could do to use a nice and simple NAS running truenas core as storage that is able to be dynamically provisioned for use by kubernetes. Now, I hear you, “what about nfs-client-provisioner”, someone less familiar with this pile of madness might exclaim. Indeed it does exist and barely work except the helm chart for it is deprecated and it does not work with kubernetes version 1.20 and later since it does not seem to be using CSI drivers.

I really tried

Months of furious and frustrating testing ensued. Not only was I trying to get applications running on kubernetes, I was also fighting with the unexpectedly complex tast of using the storage server I had available.

glusterfs

I’ve gone through basically any and all commonly suggested options for dynamically provisioned storage. Prette early on I tried glusterfs with heketi by making 3 LXC containers and mounting an nfs share to each one that would serve as the brick in the gluster volume. Suffice it to say that it didn’t work and things were getting out of hand.

longhorn

After a bit more research and I found out that longhorn, another project by the authors of k3s, used iscsi to communicate with kubernetes. That could work I thought, except there was no coherent example of how to use it without using targetd for iscsi. Longhorn was starting to look more appealing, I could just put the virtual drives of the VMs on an nfs share, run longhorn inside the VMs to pool all their storage together and call it good enough. No, I could not give up yet. A sub-optimal solution would do if there was no other way but I was convinced something more was out there. My patience and hope were running out but not empty yet.

success

That all changed in early January where during my nearly daily search for possible storage solutions I hadn’t tried, I found out about democratic-csi. This was it. Made to be used with freenas, truenas as well as DIY ZFS setups. The silver bullet was here. Just a few minutes of reading about it and writing my configs, a short(read: long) helm command later and…success! The test pod was using the newly created storageclass that was backed by the nfs share on truenas. After that I rushed to deploy my standard set of gitea, droneci and minecraft to test it out and it was working for real without a hitch.

Conclusion

This was a long journey and it isn’t coming to a close any time soon. The next step is to continuous deployment but that’s an issue for another time. To conclude as briefly as possible, I’ve no put most of the critical pieces of my homelab in a single git repo that I can use to recreate almost all of it from scratch with minimal manual intervention (there are still a few quirks like not having dynamic inventory for ansible and having to manually copy IPs but I can put that aside for now). Despite the imperfections like not having dynamic inventory for ansible or having secrets stored unencrypted in git, I’m very happy with the setup is working. If you missed them, make sure to check out the hardware updates and the first part about software changes for a better of what I’ve been up to.

Homelab Evolved

Sun, 07 Feb 2021 13:53:10 +0200

The hardware side of the changes made to my homelab were covered in a previous post where I also alluded to some software changes.

What came before

This is the first part explaining more in-depth the issues that I had and how I dealt with them. Let’s set the stage first, my homelab was working fine in many ways. It was nice and stable debian install running kvm virtual machines using qemu and libvirt with virt-manager to perform the simpler start/stop tasks. There was really nothing wrong with it, I was exploring different software, running some services on the local network and even hosting a public website on it.

Good enough is not good enough

However as someone interested in infrastructure a few things were bothering me. Sure, I was using docker-compose with traefik for my public website and services. Yes, I was using ansible to manage the configuration of most virtual machines. But “good enough” and “I guess it works” doesn’t cut it. On the upside, I really liked having ZFS snapshots on a 2x2TB mirror. The ability to go back to a point in time where things were not broken had saved me a couple times when I inevitably broke one thing or deleted a file that could be replaced but would take a while. My backups were also on that ZFS mirror, on a different dataset of course, and that also very useful. I knew I wanted ZFS and that there was much room for improvement in regards to handling how I was running services. The single docker-compose virtual machine for the public stuff and another one for local stuff were mostly adequate but the manual management was not. Not to mention that and all my precious stuff was on one box which is well below ideal.

Taking the step

One day I decided it was time to move up in the world. Despite using docker and docker-compose for years I had never dived into kubernetes because it seemed difficult and complex. New syntax, new system with its own architecture and internal structure, new workflow, new everything. I decided to set up 3 VMs and played with default k8s as well as k3s on them. Dipping my toes into it was pretty fun and since nothing depended on it I was free to wipe it and start over. The same issue kept cropping up, having to manually install debian thrice and set up kubernetes just to delete all of it and redo it was again, less than ideal Initially I started writing a playbook before finding out that someone smarter had already gone down this path. Setting up k3s-ansible was easy so there was at least kubernetes setup automation but the whole thing was too thrown together there was no shared storage. However, I was becoming more familiar with kubernetes and learning the concepts as well as how to write yaml manifests.

Conclusion

At some point it became too bothersome, a lot of manual work was involved and basic stuff like copying over ssh keys was also done mostly manually so I had to move on. While all of this was going on, I had started planning what hardware updates I would get which you can read about here: Hardware Updates 2020