Terraform on inherent site

State of Homelab 2024

Mon, 27 Jan 2025 22:38:48 +0200

When operating a homelab there is always something new to do or something waiting to be fixed. This post’s aim is to detail the state of my homelab at the end of 2024 and beginning of 2025.

Router

The previous post was about virtualizing pfSense inside ProxMox but this setup is no longer in use. In its place, there is a new system running OPNSense on bare-metal. This was done for a few reasons, the main one being that I wanted to move away from pfSense. More than that, there was a circular dependency where the ProxMox host had to be up in order for the router to function but in order for the cluster nodes to talk to each other, the router needed to be online. In practice this was not a huge issue but it bothered me since it made experimenting more difficult because recreating the cluster meant taking down the router as well. I’ve lost a few things, mainly pfblocker-ng and native tailscale integration but they’re not as big of a loss as I had anticipated. In the end I think it was worth it, over the course of the past few years the company making pfSense has acted against the best interest of pfSense users and I did not feel like using their software anymore.

Virtualization

I’m still running ProxMox, although now it is a 3-node cluster. I’ve spent a pretty significant amount of time tweaking the bootstrap process from a fresh install to having the entire homelab operational. Somewhere along the line I tried making it so non-cluster operation of multiple hosts was supported but gave up on that eventually. Each of the nodes has a 250GB SSD used as a boot drive and for node-local storage along with a 2TB HDD used as a Ceph OSD. Since the last post about using ansible and terraform with ProxMox, I’ve figured out the process of adding tools to the debian image in the virtual machine template. There is also the option to deploy using GitLab CI so just making committing and pushing to the repo is enough to create a new virtual machine. The k3s virtual machine terraform configuration was improved significantly but I’ve not adopted modules yet which seems to be the next step for organization-related improvements. I also run all software directly on top of debian virtual machines without containers to practice writing ansible. At work, our team runs everything in Docker so it could be a good idea to start moving to something similar at home for the sake of practice.

Kubernetes

The kubernetes cluster got a significant update. I’ve adopted FluxCD to a greater degree, figured out secrets storage and a pretty good structure for the YAML files. The k3s cluster has 3 control plane nodes and 3 worker nodes, one per virtualization host. I’m using longhorn for storage with plans to try and use the Ceph cluster as an additional storage class. The networking setup consists of MetalLB to have an IP in the subnet and then run ingress-nginx as the ingress controller on that IP. No plans for checking out Gateway API have been made at current but it is a good and necessary step forward in the ecosystem, I plan to move to it after things are a bit less in flux. The DNS is manual right now due to the low amount of services but I’ve tested external-dns with PiHole and I’m aware of the OPNSense webhook option, it remains to be seen what I’ll end up using. The setup needs a bit more tender love and care but it’s running well and doing what I need it to. I have plans to move more stuff to the cluster after the foundation is solid and not just have my own demo applications in there. The thought has crossed my mind to completely do away with ProxMox and then run Talos Linux bare metal on the systems but I’d be losing out on some flexibility which I don’t want to commit to right now. All in all, kubernetes is still pretty fun and the 1.30 and later releases have made good quality of life improvements.

NAS

Since I upgraded my desktop, the old one’s internals were transplanted to a different case are now a second NAS. I’m saving up to get a big external drive so I can have an extra copy of the current main NAS so I can freely mess with them to have a solid backup solution. The changes in TrueNAS Scale regarding applications has left me waiting for things to settle down before I commit to a final setup. Overall nothing exciting on this front.

Hardware

I run mostly used office desktops and a basic managed switch. I have 3 Dell Optiplex machines: 2nd gen, 3rd gen and 4th gen Intel i5 CPUs with RAM sizes ranging from 16 to 32GB each. For storage they have a 250GB SSD and a 2TB HDD as mentioned earlier. The router is a Fujitsu equivalent desktop with a 4th gen Intel Pentium and 8GB of RAM as well as a 500GB SSD. My rackmount NAS is an i5 4570 with 16GB of RAM and 4x4TB HDDs in a 3-drive ZFS RAID-Z1 plus one hot spare. My backup NAS is an i7 4770 with 32GB of RAM and 2x8TB HDDs in a 2-way ZFS mirror. The main network switch is a Netgear GS724TV4, just a layer 2 managed gigabit switch.

Conclusion

I wanted to write all of this down to have a reference for myself when I inevitably change something and it might help someone else as well. Also the blog is back(?) after more than a 2 year silence, hopefully I will not neglect it for that long again.

QEMU Guest Agent for Terraform on ProxMox

Wed, 27 Jul 2022 13:38:18 +0300

In a previous post I talked about how I set up proxmox in order to be able to provision virtual machines on it using terraform. Since then, I’ve improved the setup by adding qemu-guest-agent to the template images without needing to create them myself from scratch and I’d like to share it here.

The Problem

A lot images intended for openstack or cloud use, be they debian or ubuntu or something else, do not include qemu-guest-agent by default. This means that we either don’t get to enable that integration or have to do a complex dance of provisioning the VMs with the agent disabled, installing qemu-guest-agent, turning them off, applying a terraform plan with the agent enabled and booting them back up. Obviously that’s very tedious, error-prone and time-consuming especially when creating and destroying potentially dozens of VMs for testing. To figure out a solution, I went down a bit of a rabbit hole and considered a few solutions before settling on one so I’d like to take you through the selection process.

The Solutions

There are multiple ways to solve this problem and there isn’t really a standard go-to recommended one. One of these might work better for your situation and that’s great however I will be detailing why I didn’t choose them.

Images with qemu-guest-agent pre-installed

This was my first instinct, just find images that have it already. Unfortunately I couldn’t find a debian image that fit this criteria and overall it’s not a reliable solution, some distros might simply elect to not include it.

Terraform remote-exec provisioner

I found this great blog post by Besmir Zanaj which was really interesting and demonstrates how to use terraform to achieve post-provision VM customization. It is worth checking out however for something so basic that all virtual machines need I decided it was better to bake it into the template rather than have the action run on each VM individually. The example from the post about firewall rules is a great use-case for this functionality so it’s not to be discarded entirely.

Packer proxmox-iso Builder

Packer is another tool from Hashicorp, the same company behind terraform, that can create images to be used with proxmox and then make templates based on those images. The functionality is offered by a plugin, specifically a builder, called proxmox-iso. It didn’t appear to be a terrible choice but the configuration examples that I found relied on launching ubuntu, waiting a certain amount of time then sending keystrokes to begin a no-cloud autoinstall. This seemed to me to be very error-prone and distro-specific so I skipped it. I don’t believe it to be a complete dead-end and it’s possible there is a better way to use it for what I want but I couldn’t figure out how.

Adding qemu-guest-agent using guestfs-tools

The last thing I looked into was using guestfs-tools to modify the image before creating a proxmox template out of it. Guest-fs tools is a set of tools for modifying virtual machine disk images that works on a lot of distros. After lots of searching, I came across an awesome blog post by Austin Pivarnik which tackled basically the same problem. As detailed in the post itself, on debian-based systems which luckily proxmox is, we can install libguestfs-tools and then use the virt-customize command to alter the contents of a virtual machine image. How I added this to my setup:

add libguestfs-tools to the list of packages installed by my proxmox setup ansible playbook
add a task (adapted for all VM images) to the same playbook after downloading the image but before creating the template
run playbook
ta-da! You can see the changes I made in this commit where I also adjusted my terraform configuration and a couple other small things. I’ve been very happy with this so far and aside from showing the guest IP in the proxmox web interface as well as allowing smooth shutdown it has also improved provisioning times.

Conclusion

Having good templates for virtual machines on a virtualization platform is very important for automation. While it took quite a long time for me to find a suitable solution I am quite happy with where I ended up and the improvement to my homelab. Thank you for reading and I hope you learned something new.

Ansible, Terraform and Proxmox

Fri, 24 Jun 2022 23:24:57 +0300

In the continuing quest to improve my homelab I’ve written a few ansible playbooks as well as terraform configuration to automate a lot of tasks. The initial setup of proxmox is handled by ansible, the provisioning of virtual machines is handled by terraform and the installation of k3s on the VMs is done by ansible.

Proxmox Setup with Ansible

Before getting to use all the shiny devops with a selfhosted solution, we need to do a bit of setup first. This isn’t perfect since it’s just ansible running a bash script but it gets the job done. The other issue,, is that a lot of cloud images don’t include qemu-guest-agent and other packages by default. I have a few ideas how to solve that (check out this TODO.md for more info) but haven’t tested out anything yet. Since qemu-guest-agent and things like nfs-common are required on most VMs, a lot of redundant package installation happens but such is life for now. With that out of the way, let’s start with how I configure proxmox.

Host

While proxmox has a lot of things ready to go out of the box, a bit of configuration is always required and even more so in this case. The goal of the following steps is to turn our hosts into capable, cloud-like, targets for provisioning resources using terraform.

Repository

Initially the enterprise repository which we don’t have a license to access is enabled. To fix this, I delete the file in /etc/apt/sources.list.d that contains it and add a file with the no-subscription repository. This might not be necessary since the lae.proxmox ansible role can do essentially the same thing and many more. Here are the ansible tasks:

    - name: remove enterprise repo
      file:
        path: /etc/apt/sources.list.d/pve-enterprise.list
        state: absent

    - name: add no-subscription repo
      copy:
        src: pve-no-sub.list
        dest: /etc/apt/sources.list.d/pve-no-subscription.list
        mode: '0644'
        owner: root
        group: root

Template VM

The terraform provider for proxmox can work with cloud-init which is a great vendor-agnostic way of configuring the basics (username, password, IP addressing, DNS, SSH keys) in a virtualized environment. This is achieved by having a virtual disk or virtual cd-rom that mounts inside the VM and provides this information to the guest.

VM Image

Since software is not always compatible with the latest version of a linux distribution, in this case debian, I keep the last two versions around. The first step is to download the images:

    - name: get debian 10 cloud image
      get_url:
        url: https://cloud.debian.org/images/cloud/buster/20211011-792/debian-10-generic-amd64-20211011-792.qcow2
        dest: /root/debian-10-openstack-amd64.qcow2
        checksum: sha512:f3dac13104b4e28eb62c46cbbb1e30fc9c792834500f9101d477c19c258c31ff04850933ee0b3e63236eca38c854447d95a0ab45163c4a3fccf05f9dd95601a5
        mode: '0644'
        owner: root
        group: root

    - name: get debian 11 cloud image
      get_url:
        url: https://cloud.debian.org/images/cloud/bullseye/20220121-894/debian-11-generic-amd64-20220121-894.qcow2
        dest: /root/debian-11-openstack-amd64.qcow2
        checksum: sha512:0948dc56b4834a7755e4eae7d5532e138b90484a949161ffe9fc6894c7a14b1bd32ebf96fa3f3d03d498fe7ee125f8014f31bfab5825915a93de9330df814f7b
        mode: '0644'
        owner: root
        group: root

Template Script

As mentioned in the beginning I’d like to automate this with ansible but this is what I have for now:

if [ -z "$1" ]; then
    VMID=9001
else
    VMID="$1"
fi

if [ -z "$2" ]; then
    VMNAME="debian-tmpl"
else
    VMNAME="$2"
fi

VMEXISTS="$(qm list | grep $VMID)"

if [ "$VMEXISTS" ]; then
    echo "VM with ${VMID} already exists"
else
    qm create "${VMID}" -name "${VMNAME}" -memory 1024 -net0 virtio,bridge=vmbr0 -cores 1 -sockets 1 -cpu cputype=kvm64 -description "debbie image" -kvm 1 -numa 1
    qm importdisk "${VMID}" debian-10-openstack-amd64.qcow2 local-zfs
    qm set "${VMID}" -scsihw virtio-scsi-pci -virtio0 local-zfs:vm-"${VMID}"-disk-0
    qm set "${VMID}" -serial0 socket
    qm set "${VMID}" -boot c -bootdisk virtio0
#   qm set "${VMID}" -agent 1 #disabled since VM does not initially have qga installed
    qm set "${VMID}" -hotplug disk,network,usb,memory,cpu
    qm set "${VMID}" -vcpus 1
    qm set "${VMID}" -vga qxl
    qm set "${VMID}" -name "${VMNAME}"
    qm set "${VMID}" -ide2 local-zfs:cloudinit
    qm template "${VMID}"
fi

This works well enough for first-time setup and with a few additions could become idempotent. The above script and its debian 11 equivalent are copied over to the proxmox hosts and executed by the following ansible tasks:

    - name: get debian 10 template vm script
      copy:
        src: setup-debian-10-cloudinit.sh
        dest: /root/setup-debian-10-template.sh
        mode: '0755'
        owner: root
        group: root

    - name: run debian 10 template vm script
      shell:
        cmd: /root/setup-debian-10-template.sh

    - name: get debian 11 template vm script
      copy:
        src: setup-debian-11-cloudinit.sh
        dest: /root/setup-debian-11-template.sh
        mode: '0755'
        owner: root
        group: root

    - name: run debian 11 template vm script
      shell:
        cmd: /root/setup-debian-11-template.sh

Terraform Setup

With proxmox ready, let’s move on to the terraform setup. If you haven’t used it before, terraform is a declarative way of describing our infrastructure. It can manage virtual machines, DNS records, object storage buckets, kubernetes resources and many more things with the appropriate providers.

Provider

Providers in terraform are plugins that enable terraform to manage resources that it doesn’t support out of the box. In this case, we want to manage proxmox virtual machines and containers but it could be other things like a router running pfsense or vyos. In previous versions, the provider would need to be installed manually but nowadays we only need to create a main.tf with the provider listed and upon running terraform init, the provider will be downloaded automatically.

Basic `main.tf`

Let’s create that file then. Open your favorite editor, paste the following and replace the IP as well as the password with the ones applicable to your setup:

terraform {
	required_providers {
		proxmox = {
			source = "Telmate/proxmox"
			version = ">=2.9.5"
		}
	}
}

provider "proxmox" {
	alias = "pve"
	pm_tls_insecure = true
	pm_api_url = "https://192.168.70.1:8006/api2/json" #Replace IP
	pm_password = "password123" #Replace password
	pm_user = "root@pam"
}

Save the file and then run terraform init. You should see the following output (the version might be different):

Initializing the backend...

Initializing provider plugins...
- Finding telmate/proxmox versions matching ">= 2.9.5"...
- Installing telmate/proxmox v2.9.10...
- Installed telmate/proxmox v2.9.10 (self-signed, key ID A9EBBE091B35AFCE)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

And ta-da! We’ve installed the provider and are ready to start creating resources.

Creating a test VM

The above file won’t do anything so we’ll declare a virtual machine and then tell terraform to create it. The example below is specific to my setup so you probably want to modify it before using but here it is:

terraform {
	required_providers {
		proxmox = {
			source = "Telmate/proxmox"
			version = ">=2.9.5"
		}
	}
}

provider "proxmox" {
	alias = "pve"
	pm_tls_insecure = true
	pm_api_url = "https://192.168.70.1:8006/api2/json" #Replace IP
	pm_password = "password123" #Replace password
	pm_user = "root@pam"
}

resource "proxmox_vm_qemu" "proxmox_test_vm" {
	provider = proxmox.pve
	name = "deb-10-test"
	target_node = "pve"

	/* change to debian-templ for 11 */
	clone = "debian-tmpl"
	os_type = "cloud-init"
	cores = 2
	sockets = 1
	cpu = "host"
	memory = 2048
	scsihw = "virtio-scsi-pci"
	bootdisk = "virtio0"
	agent = 0

	disk {
		size = "30G"
		type = "virtio"
		storage = "local-zfs"
	}

	network {
		model = "virtio"
		bridge = "vmbr0"
	}

	ipconfig0 = "ip=192.168.50.25/16,gw=192.168.0.1"

	/*workstation SSH key*/
	sshkeys = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5jzKi37jm3517bqThbw+7LR/GXm3qC6Az5F+ZUa36vYM7Ygk2K5bWcFIL2YUCrkL5jfSsvoowONjCAxyuoyxtW4MJxnQLyq4u4yDsRC7YvBPAKZUYaHwnbkCfDs5a75dEFOoDxCA0DY2GrhqzBndaTcCfl0fZ4vN+9LcKOb1dSKiHeHvsh35YNtwntbL21meo+hiycUEgGwNe9/4kxKpdGTr7HvbeX2Fjm/UZBZIJKVcGop/3gCHXYnKH+OY5zc8cmt9Jg4CIwEqrSKeOX0bE8LSPRpVRXH4v8OcMaMei/HQejlH8NBwybEdJ4mhl8vHaFEjDbIWoOujmiRQF2263 angle@puddle"

	/*required otherwise the state always appears modified*/
	lifecycle {
		ignore_changes = [
			cipassword,
			network,
			desc,
		]
	}
}

After saving the file, we can run terraform plan -out create-test-vm.plan to calculate the changes required and save them to a file. Following that, we can run terraform apply create-test-vm.plan to apply the changes to our infrastructure. Some output will appear while the template is cloned, the VM settings are adjusted and the cloud-init configuration is put into the drive. In the end, you should have a brand new VM created by terraform!

Creating VMs for k3s

The above example is realistic but what if we need 5 similar VMs that could be used as kubernetes nodes? Conveniently, terraform provides a count variable that we can use to slightly differ settings where needed. Here is the resource that provisions the virtual machines that I run a k3s cluster on:

resource "proxmox_vm_qemu" "proxmox_vm_k3s" {
	provider = proxmox.pve
	count = 5
	name = "deb-k3s-${count.index + 1}"
	target_node = "pve"

	clone = "debian-tmpl"
	os_type = "cloud-init"
	cores = 2
	sockets = 1
	cpu = "host"
	memory = 3084
	scsihw = "virtio-scsi-pci"
	bootdisk = "virtio0"
	agent = 0
	onboot = true

	disk {
		size = "30G"
		type = "virtio"
		storage = "local-zfs"
	}

	network {
		model = "virtio"
		bridge = "vmbr0"
	}

	ipconfig0 = "ip=192.168.15.5${count.index + 1}/16,gw=192.168.0.1"

	sshkeys = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5jzKi37jm3517bqThbw+7LR/GXm3qC6Az5F+ZUa36vYM7Ygk2K5bWcFIL2YUCrkL5jfSsvoowONjCAxyuoyxtW4MJxnQLyq4u4yDsRC7YvBPAKZUYaHwnbkCfDs5a75dEFOoDxCA0DY2GrhqzBndaTcCfl0fZ4vN+9LcKOb1dSKiHeHvsh35YNtwntbL21meo+hiycUEgGwNe9/4kxKpdGTr7HvbeX2Fjm/UZBZIJKVcGop/3gCHXYnKH+OY5zc8cmt9Jg4CIwEqrSKeOX0bE8LSPRpVRXH4v8OcMaMei/HQejlH8NBwybEdJ4mhl8vHaFEjDbIWoOujmiRQF2263 angle@puddle"

	lifecycle {
		ignore_changes = [
			cipassword,
			network,
			desc,
		]
	}
}

There aren’t many changes from the previous example, the name and the IP address use the index in order to avoid 5 VMs having the same name and IP address and that’s it!

Conclusion

Hopefully this gives you inspiration for some awesome things you can do to automate your infrastructure and make working in a virtualized environment easier. Thank you for reading.

Homelab Current Form

Sun, 07 Feb 2021 14:35:32 +0200

The first part of the software changes I made was covered in the first part where I explained how and why I started going down this path.

Software choices

This whole journey was about making my home infrastructure better. Part of that was about having a way to more easily create the foundation on which kubernetes would run on, as well as describing my setup as code. Proxmox is a beast in its own right. Features include being able to manage kvm virtual machines and lxc containers, an API that can be used to interact with it programmatically and most importantly being open source. While researching what automation tools could interact with proxmox, I found a community-made terraform provider as well as a set of two ansible community modules (kvm and lxc). After thinking about it a little bit, I wanted to use terraform in a non-cloud environment but also gather some knowledge about it so that’s what I decided to use. I can have my provisioning requirements in a file that terraform understands and store it in git which is exactly what I was going for. Ansible is still in the picture since it is used to set up and configure the debian environment inside the virtual machines. Now that the hypervisor part is covered, let’s move on to running services.

Making it store is harder than making it run

As mentioned in the first part about software changes, I was learning kubernetes and now I could not only wipe and recreate the cluster itself but also the entire virtual machines that it was running on. Some time after starting to use proxmox with terraform I attempted to use rancher to manage kubernetes but ended up ditching it due to various problems with running even basic stuff on it (very likely that it’s a case of PEBCAK, I don’t think rancher is terrible or anything like that).

The problem

However the problem of shared storage continued to taunt me for the following months. There was seemingly nothing that a simple fella could do to use a nice and simple NAS running truenas core as storage that is able to be dynamically provisioned for use by kubernetes. Now, I hear you, “what about nfs-client-provisioner”, someone less familiar with this pile of madness might exclaim. Indeed it does exist and barely work except the helm chart for it is deprecated and it does not work with kubernetes version 1.20 and later since it does not seem to be using CSI drivers.

I really tried

Months of furious and frustrating testing ensued. Not only was I trying to get applications running on kubernetes, I was also fighting with the unexpectedly complex tast of using the storage server I had available.

glusterfs

I’ve gone through basically any and all commonly suggested options for dynamically provisioned storage. Prette early on I tried glusterfs with heketi by making 3 LXC containers and mounting an nfs share to each one that would serve as the brick in the gluster volume. Suffice it to say that it didn’t work and things were getting out of hand.

longhorn

After a bit more research and I found out that longhorn, another project by the authors of k3s, used iscsi to communicate with kubernetes. That could work I thought, except there was no coherent example of how to use it without using targetd for iscsi. Longhorn was starting to look more appealing, I could just put the virtual drives of the VMs on an nfs share, run longhorn inside the VMs to pool all their storage together and call it good enough. No, I could not give up yet. A sub-optimal solution would do if there was no other way but I was convinced something more was out there. My patience and hope were running out but not empty yet.

success

That all changed in early January where during my nearly daily search for possible storage solutions I hadn’t tried, I found out about democratic-csi. This was it. Made to be used with freenas, truenas as well as DIY ZFS setups. The silver bullet was here. Just a few minutes of reading about it and writing my configs, a short(read: long) helm command later and…success! The test pod was using the newly created storageclass that was backed by the nfs share on truenas. After that I rushed to deploy my standard set of gitea, droneci and minecraft to test it out and it was working for real without a hitch.

Conclusion

This was a long journey and it isn’t coming to a close any time soon. The next step is to continuous deployment but that’s an issue for another time. To conclude as briefly as possible, I’ve no put most of the critical pieces of my homelab in a single git repo that I can use to recreate almost all of it from scratch with minimal manual intervention (there are still a few quirks like not having dynamic inventory for ansible and having to manually copy IPs but I can put that aside for now). Despite the imperfections like not having dynamic inventory for ansible or having secrets stored unencrypted in git, I’m very happy with the setup is working. If you missed them, make sure to check out the hardware updates and the first part about software changes for a better of what I’ve been up to.

Terraform on inherent site

State of Homelab 2024

Router

Virtualization

Kubernetes

NAS

Hardware

Conclusion

QEMU Guest Agent for Terraform on ProxMox

The Problem

The Solutions

Images with qemu-guest-agent pre-installed

Terraform remote-exec provisioner

Packer proxmox-iso Builder

Adding qemu-guest-agent using guestfs-tools

Conclusion

Ansible, Terraform and Proxmox

Proxmox Setup with Ansible

Host

Repository

Template VM

VM Image

Template Script

Terraform Setup

Provider

Basic main.tf

Creating a test VM

Creating VMs for k3s

Conclusion

Homelab Current Form

Software choices

Making it store is harder than making it run

The problem

I really tried

glusterfs

longhorn

success

Conclusion

Basic `main.tf`