In the continuing quest to improve my homelab I’ve written a few ansible playbooks as well as terraform configuration to automate a lot of tasks.
The initial setup of proxmox is handled by ansible, the provisioning of virtual machines is handled by terraform and the installation of k3s on the VMs is done by ansible.
Proxmox Setup with Ansible
Before getting to use all the shiny devops with a selfhosted solution, we need to do a bit of setup first.
This isn’t perfect since it’s just ansible running a bash script but it gets the job done.
The other issue,, is that a lot of cloud images don’t include qemu-guest-agent and other packages by default.
I have a few ideas how to solve that (check out this TODO.md for more info) but haven’t tested out anything yet.
Since qemu-guest-agent and things like nfs-common are required on most VMs, a lot of redundant package installation happens but such is life for now.
With that out of the way, let’s start with how I configure proxmox.
While proxmox has a lot of things ready to go out of the box, a bit of configuration is always required and even more so in this case.
The goal of the following steps is to turn our hosts into capable, cloud-like, targets for provisioning resources using terraform.
Initially the enterprise repository which we don’t have a license to access is enabled.
To fix this, I delete the file in /etc/apt/sources.list.d that contains it and add a file with the no-subscription repository.
This might not be necessary since the lae.proxmox ansible role can do essentially the same thing and many more.
Here are the ansible tasks:
The terraform provider for proxmox can work with cloud-init which is a great vendor-agnostic way of configuring the basics (username, password, IP addressing, DNS, SSH keys) in a virtualized environment.
This is achieved by having a virtual disk or virtual cd-rom that mounts inside the VM and provides this information to the guest.
Since software is not always compatible with the latest version of a linux distribution, in this case debian, I keep the last two versions around.
The first step is to download the images:
This works well enough for first-time setup and with a few additions could become idempotent.
The above script and its debian 11 equivalent are copied over to the proxmox hosts and executed by the following ansible tasks:
- name:get debian 10 template vm scriptcopy:src:setup-debian-10-cloudinit.shdest:/root/setup-debian-10-template.shmode:'0755'owner:rootgroup:root- name:run debian 10 template vm scriptshell:cmd:/root/setup-debian-10-template.sh- name:get debian 11 template vm scriptcopy:src:setup-debian-11-cloudinit.shdest:/root/setup-debian-11-template.shmode:'0755'owner:rootgroup:root- name:run debian 11 template vm scriptshell:cmd:/root/setup-debian-11-template.sh
With proxmox ready, let’s move on to the terraform setup.
If you haven’t used it before, terraform is a declarative way of describing our infrastructure.
It can manage virtual machines, DNS records, object storage buckets, kubernetes resources and many more things with the appropriate providers.
Providers in terraform are plugins that enable terraform to manage resources that it doesn’t support out of the box.
In this case, we want to manage proxmox virtual machines and containers but it could be other things like a router running pfsense or vyos.
In previous versions, the provider would need to be installed manually but nowadays we only need to create a main.tf with the provider listed and upon running terraform init, the provider will be downloaded automatically.
Let’s create that file then.
Open your favorite editor, paste the following and replace the IP as well as the password with the ones applicable to your setup:
Initializing the backend...
Initializing provider plugins...
- Finding telmate/proxmox versions matching ">= 2.9.5"...
- Installing telmate/proxmox v2.9.10...
- Installed telmate/proxmox v2.9.10 (self-signed, key ID A9EBBE091B35AFCE)
Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
And ta-da! We’ve installed the provider and are ready to start creating resources.
Creating a test VM
The above file won’t do anything so we’ll declare a virtual machine and then tell terraform to create it.
The example below is specific to my setup so you probably want to modify it before using but here it is:
After saving the file, we can run terraform plan -out create-test-vm.plan to calculate the changes required and save them to a file.
Following that, we can run terraform apply create-test-vm.plan to apply the changes to our infrastructure.
Some output will appear while the template is cloned, the VM settings are adjusted and the cloud-init configuration is put into the drive.
In the end, you should have a brand new VM created by terraform!
Creating VMs for k3s
The above example is realistic but what if we need 5 similar VMs that could be used as kubernetes nodes?
Conveniently, terraform provides a count variable that we can use to slightly differ settings where needed.
Here is the resource that provisions the virtual machines that I run a k3s cluster on: