Sysadmin on inherent site

How This Website Is Made

Sat, 20 Nov 2021 20:56:19 +0200

Intro

A significant part of having my own site is the way it’s made behind the scenes. This has changed in a couple different ways for the duration of its existence and in this post I’ll try to explain how and why.

Beginning

When I first bought inherently.xyz to host my own website I wanted my site to be built in the way that I’d like other sites to be built. The main requirements were:

a theme that was mine
basically no javascript
nothing loading from third-party places
code that is easy to understand
readable in the devtools
reasonably small and fast
able to be read on a lot of devices and browsers
over the top way to host it

I don’t think I have a repository where I developed it but this is an archive of some version of it. I’ll retell the story of its existence so you don’t have to look at the source but it’s there if you want to take a look.

Evaluation

In my humble opinion I did fairly well to cover the requirements I had but not as well as I could have. However not all was great. For one, I had used bootstrap, the CSS framework, to make the site. Because I used the navbar hamburger/button/dropdown thing it required some javascript/jquery which was unfortunate. It was my goal to get rid of this but from reading a github issue about it the team behind bootstrap did not feel like providing that functionality through a CSS trick was something they were willing to do.

The theme was mostly mine outside of boostrap and it was adapted from someone else’s CSS colorscheme. As for the code itself, every page consisted of hand-written HTML so that part was easy to read but the minified bootstrap files, aside from being pretty damn big, were unreadable. At least the bootstrap files were hosted on my own server and no third-party anything loading so I can check that off the list. Additionally, to make the site I had created a file called base.html containing the basic template of a page which I would manually copy to create a new page and then rsync the files from my computer to the server. Surprisingly enough, this worked pretty well although changing code that touched multiple pages like the navigation was somewhat cumbersome. Reading the website through a lesser known web browser like netsurf, surf or any text-mode web browser was possible so that requirement was basically 100% satisfied.

Since the beginning of this site I’ve been using docker-compose to host it. For this iteration I was using a simple nginx image and mapping the directory on the filesystem to /usr/share/nginx/html inside the container. Traefik was running in front of it acting as a reverse proxy and to support https. Overkill for a site that has handwritten HTML? Check.

Improvements

There was quite a bit of way to go as you can gather from the above. The main pain points were the existence of javascript causing the navigation dropdown to not work on mobile when javascript was disabled, the performance metrics related to using bootstrap in general, dependency on bootstrap and lack of major customization from me. The raw HTML workflow was mildly annoying but it never stopped me from writing anything.

Solutions

I spent quite a long time trying to find a workaround for the nav dropdown and eventually stumbled into a solution in a post I can’t find anymore, I’ll update this if I do. That worked fine enough and allowed me to get rid of the jquery dependency that, funnily enough, had a vulnerability announced about it while I was rewriting my nav code. Bootstrap was also slowing down everything according to all benchmarks I ran.

This was all fairly fine for quite a while. I kept working on the site, writing stuff and having my own little space on the internet. The curse of being into tech and knowing how to make a website even as basic as this was that I wanted to rewrite it. That’s when I started exploring and found out about hugo, a fast static site generator (SSG for short) written in Go. Given that I was interested in Go at the time of this rewrite and that I knew how the templates worked more or less I decided to give it a try.

Hugo

My first idea was to just port it to hugo and then improve it and quickly realized what a mistake that would be. Why do one thing you’ve never done before if you can do two, right? Hugo is a static site generator meaning that it creates the files required for the site once and that’s it, the site is not dynamically put together on the client. As the author, you mostly write markdown files that are turned into html at build time and then you can put those html files anywhere you’d like. This is what the site uses to this day and you find the source code for it here.

Transitioning

The real work started when I decided I’d make my own theme too. I didn’t just have to move my content and page structure to it but write some nice fresh CSS too since I decided I’ve had enough of bootstrap.

Directory Structure

The first steps was examining the directory structure and finding the most low level template file which is layouts/_default/baseof.html and making my way up. Inside there I defined the very basics of what the HTML would look like, imported my CSS file which at this point had about 5 lines in it and moved on to the other fundamental pages. These are layouts/_default/single.html and layouts/_default/list.html, the former being about a “one of it’s kind” page and the latter being an aggregation. The example here is a blog post and the list of all the blog posts. They are based on baseof.html and in my case are very short, consisting only of the barebones required to implement these types of pages. This is because they are overwritten by category-specific single.html and list.html which is how I make blog posts have a table of contents and how I make the blog section look different than the tutorials section.

Layouts

Speaking of sections, the first thing I got up and running was the blog since that was the easiest one with lots of documentation and examples. The second priority was the front page which turned out to be a little bit more complicated. You put the content for it inside content/_index.md and the layout-related stuff inside layouts/index.html. In hindsight it’s fairly intuitive but it took a few minutes of digging when I was less familiar. For each sub-category you can basically include no layout files in which case the list.html and single.html from layouts/_default will be used or you can write ones that will override them. As an example, in layouts/tutorials I have both a list.html and a single.html to build the tutorial-related pages in their own way.

Layouts are pretty cool and probably my favorite thing is partial layouts. These allow you to put a snippet in a file, let’s say footer.html and then include it using {{ partial "footer" . }} in another template. One other example is the contact form, I just made a layouts/contact/list.html that has the line {{ partial "comments.html" . }} and this way I can keep the template cleaner and allow reusing that comment submission form elsewhere too. So I just went around creating my categories, writing specific list or single templates for them if required and throwing some pieces of reusable code in partial layout templates.

CSS

The most involved part was by far the CSS. I have to say that if it’s not obvious by now I’m not a “frontend person” per se, I just want something that looks clean, fast and readable. As a disclaimer, I’m not a fan of the size of the fonts which are by far the biggest element right now but I’m a huge fan of the fira font family (with the exception of fira code cause I hate ligatures) but they work well enough. It took the majority of the time and had the most updates, tweaks and fixes by far compared to anything else. Essentially nothing from the original CSS is the same except the colors which was a long process but a necessary one to get where I wanted to be. In the end I’m satisfied with it, I think it looks fairly unique and serves its purpose well while coming in at under 200 lines (it is way less in reality but the number balloons up because of the way I format the code).

Docker

This time I went with a custom docker image so I could have more control and no filesystem bind mounts. The main problem is that the hugo team does not publish an official docker image so I had to use the following to get hugo installed in the container in order to generate the site (most current Dockerfile is in the git repository, check there first before copying):

FROM golang:1.16 as buildsite
ENV CGO_ENABLED 0
WORKDIR /go/src
RUN git clone https://github.com/gohugoio/hugo.git
WORKDIR /go/src/hugo
RUN go install -v
WORKDIR /go/src/inheresite-hugo
COPY . .
RUN hugo

FROM nginx:alpine
COPY default.conf /etc/nginx/conf.d/
COPY --from=buildsite /go/src/inheresite-hugo/public /usr/share/nginx/html

This means I compile hugo every time I want to upload a post but it goes fairly fast and gives a bit of time to reflect on what I’ve written. If this is not for you, use either an ubuntu image and install it with apt if you don’t need the latest features or an arch one if you need the latest release for some reason. Using third-party images is a last resort but those are technically also usable for this purpose however I’d advise against using images from random people.

Impressions

After a long while with hugo I think I’ll be sticking with it. It allows for some pretty clean code, I’m partial to Go templating and it has served me well. I’d like to look into some small improvements such as merging tutorials into blog and just displaying them in a different way but that’s a topic for another day. The full server-side syntax highlighting is awesome too, I thought I’d need to cave and use some javascript thing to do it but thankfully it worked out great and I found a code theme that matches the rest of the site. If you want to get started with it, just make sure you know some basic HTML and have some patience, the rest can be figured out in an evening.

Conclusion

The site you’re currently looking at has been a while in the making and there are still a couple items on my TODO.md for it but I’m happy enough with it for now. You might want to check out the comment microservice I wrote for use with the comment form mentioned a bit above, it’s far from perfect but it gets the job done. Thank you for reading this, I hope you enjoyed it and maybe even learned something.

Flux Update

Wed, 25 Aug 2021 00:44:34 +0300

I wrote about my journey of choosing a continuous deployment tool and why I ultimately ended up using flux for my homelab in this post. It took quite a bit of reasearch and if you’re in a similar position as I was it might prove to be helpful.

Long(ish)-term experience

I’ve been using it for about 5 months now, the first post was written about one month of use. At that point I had been tinkering with it and attempting to get all the stuff I wanted to be deployed by it. Some things were simple, others not so straight-forward.

Successes

Deploying raw manifest applications

Deploying raw kubernetes manifests was really easy. Since I’m somewhat on the fence about Helm, I prefer using normal kubernetes manifests where possible. This meant that for my use case mostly everything was smooth sailing after figuring out how to generate a kustomization yaml using the command-line tool.

Command-line tool

Speaking of that, flux has a command-line tool to interact with the controllers running inside the cluster. It’s also the way you can initialize a repository to be used as the place for flux files to be stored. It can also be used to install flux to the cluster if the repository already exists.

Upgrading

When a new version of flux comes out, you can upgrade and it will carry over to the cluster. It’s really easy, it just upgrades its own files so when those are committed it will pull in the new version of the controllers.

Trouble

Aggregation yaml

For starters here is a minor one that took way too long to diagnose. I forget exactly what this is called but essentially it is a yaml file says run deal with the listed resources. Imagine the following directory contents:

drwxrwxr-x 4 angle angle 4,0K Αυγ  25 01:23 .
drwxrwxr-x 5 angle angle 4,0K Ιουλ 21 03:42 ..
drwxrwxr-x 2 angle angle 4,0K Αυγ  13 15:55 helmrepos
-rw-rw-r-- 1 angle angle  219 Ιουλ 21 03:20 infra-source.yml
-rw-rw-r-- 1 angle angle  126 Αυγ  12 11:45 kustomization.yaml
drwxrwxr-x 2 angle angle 4,0K Αυγ  13 15:56 storage

The kustomization.yaml file is as follows:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - infra-source.yml
  - helmrepos
  - storage

So it’s a way to more easily include stuff. The issue I had is that I named the file kustomization.yml instead of kustomization.yaml which for some reason isn’t supported. I believe this is a bug and not intended behavior but it was still frustrating to find and fix.

Logs

Now on to more substantial issues. While troubleshooting I got a bit frustrated with how flux displays errors. The messages are fairly non-descript and there isn’t a lot of guidance for how to fix them.

Conclusion

Despite a couple rough edges, flux has been working incredibly well. Even before I got it fully functioning, it was amazing to be able to just install it to a newly creted cluster and have all my applications running just like that. It’s almost magical and I’ve been enjoying it quite a lot. If you aren’t using a continuous deployment tool, feel free to give flux a try or read the previous post to learn about some other options

Droning On

Sat, 01 May 2021 18:54:59 +0300

I described previously how the deployment of applications is automated in my homelab but didn’t touch upon how I test my own code. The topic came up when I started trying to simulate a development pipeline who’s first step after committing code is to be run through the continuous integration system. What should that system be though? In this post, I’ll take you through the things I tried first and then why I went with drone in the end.

Choose a Fighter

My quest is to ultimately end up with the skills to set up kubernetes so that it can be deployed to everything from a DIY NAS and a stack of NUCs to a warehouse full of servers backed by distributed storage. I’m obviously more towards the first camp so that’s where my journey is at. With that said, my goal is to have software small and lightweight enough that it can run on the former setup easily so that there can be multiple instances of it in the latter form. The current mental model I have is developer organization with not much more than 3 NUCs + 1 NAS per team without losing the ability for continuous integration and continuous deployment. Therefore the landscape needs to be explored and the obvious titan of the old world is jenkins with many newcomers eager to take its place. The obvious caveats still apply, it must be open-source, able to be self-hosted, support for gitea is needed and being able to run well on kubernetes is a foregone conclusion.

Jenkins

It should need no introduction, most developers are familiar with the trusty crusty old butler software. Jenkins is probably one of the most prominent pieces of CI/CD software that was pushed to the forefront when the push for more automation and increased code quality really happened. Sadly there are a few reasons that annoyed me from the get-go. First, the unlocking procedure requires you to kubectl logs <podname> in order to find the unlock key which I understand but nonetheless find annoying. Second, the Jenkinsfile has its own domain-specific language (DSL for short) that I find unappealing even though you won’t catch me praising YAML’s lack of curly braces. I begrudgingly went though it and wrote a Jenkinsfile to give it a fair chance but it wasn’t a nice experience. Next, a single instance uses too much ram to the point that it can barely fit in my k3s VMs which use 2.5GB of ram each together with other software. This is an issue because I like running at least 2 instances of most applications to be sure that everything works correctly when scaling up. Last but not least, it’s a 2-in-1 solution which I don’t find appealing as far as architecture and since I was planning to find a standalone piece of software to do CI and another one to handle CD, it was a hard sell from the start. As a redeeming quality it has an official kubernetes pipeline plugin which is nice so they’ve at least put thought into this use case. Ultimately it wasn’t the right fit for me so it was time to move on.

Drone

While pondering this I remembered an old post from one of the blogs I read, specifically this one. I said what the hell, why not and started looking its website. Drone is a very nice and simple continuous integration tool. From the start it seemed like it would be a good pick but you never know until you try. I wrote some kubernetes manifests, got angry until I got them to work and a half hour later it was hooked up to my local gitea instance and ready to use. I added a .drone.yml to one of my repos, pushed the change, activated the repo from the drone dashboard, made another change to the source code, pushed it and waited to see what happened. Instantly drone received the webhook event and got to work. The dashboard is pretty simple (and sadly lacks a dark mode) so you just hit the thing you want to look at, click on the build and see it happen. It supports different things running at the same time like databases if you want your tests to include that which can be run either as what drone calls services aka run in the background from the beginning or if you want an ordered dependency you can use detached steps which is especially useful in microservices. My only real complaint aside from the lack of dark mode is the inability to trigger a build manually without pushing but that’s only an issue when you first activate a repository inside drone’s dashboard. All in all, it has been a joy to use, most of the projects I am currently developing have a .drone.yml and get built at least 3 times a day when gitea mirrors the gitlab or github repo they’re mainly hosted at.

Conclusion

This wasn’t as big of a journey as finding a continuous development tool because really I only found the two aforementioned options that had the basics I was looking for and one of them didn’t really fit the bill. I’m not huge into software development, it’s mostly a hobby for me as might be obvious by now, since my main concern is infrastructure-related stuff. With that said it’s useful to have the skills to build an application that can communicate with the kubernets api to automate something or write an ansible module and most of all it allows me to see the developer perspective. I haven’t started looking into SOPS or renovate bot or the k3s upgrade controller or velero but I hope to get the chance to do so in the future. Thank you for reading, I hope you enjoyed it and maybe learned something.

Fluxing My Cluster

Sat, 17 Apr 2021 17:24:22 +0300

It’s no secret that I’m a fan of automation and making life easier (even if I made it harder in the first place). One of the issues I’ve been having in my homelab is dealing with deploying stuff to kubernetes. Initially I wanted to just add a plugin to drone and be done with it. However, that didn’t really pan out which ended up being to my benefit since I discovered flux.

The problem

The cluster in its current state is a little sad because I end up doing too many things manually. Whenever something changes it’s time to bring out ye old kubectl apply -f in a local clone of the repo and that doesn’t spark joy. This was obviously bad form and couldn’t continue. What you also have to keep in mind is that I’ve chosen a 3rd-party storage plugin, democratic-csi and assume it’s the default storageclass in the cluster. This means that on a new cluster it’s required that before anything else, that one is added as a storageclass and set as default and local-path is removed from being a default. After that, the applications can be deployed which includes a mix of stuff I’ve written for the purposes of testing as well as 3rd-party software. Now that the stage has been set, let’s see what I tried

Solutions

Drone CI plugin

Initially the idea was that since I’ve alrady solved this for code using drone for CI (this will be discussed in a later post), I could just use a plugin and be done. However, my experience with writing helm charts is limited and the only official plugin for deploying to kubernetes is for helm charts, not kubernetes manifests. Furthermore, as I mentioned previously, not all software deployed on the cluster is something that I have written so even if there was an official kubernetes manifest plugin it wouldn’t cover 3rd-party software. At this point, I was thinking of finding some all-inclusive CI/CD software that would cover everything I wanted to do. It took no more than 10 seconds of thinking to realize that I was looking for non-modular software and discard that idea.

Standalone Continuous Delivery

This was the obvious choice but the software in this category is plentiful. The main competitors were flux, argo and tekton. Their designs differ quite a bit so there is going to be quite a bit of opinion in the following analysis so just keep that in mind. The few things they had in common were that they were all advertised as cloud-native with support for kubernetes, had a cli and were written in Go.

ArgoCD

I started with argo because one of the people that wrote a really good blog post on using kubernetes at home mentioned that they use it so I thought I’d try it first. The installation process required a massive yaml file that you just curl | kubectl apply -f which I wasn’t a big fan of but alright whatever. It also had a weird unlock procedure like jenkins where you have to find a token generated at runtime to unlock it but that’s okay, just an one-time setup thing. After spending a couple evenings with it, I wasn’t satisfied with the experience. The docs weren’t really good at explaining all the argo custom resources and getting a basic single-pod application running took me more than 2 hours. It was doable but this was going to be something I had to do for every application running on my cluster and the prospect didn’t seem appealing. I’m sure with more time I could maybe become more familiar and eventually warm up to it however first impressions were bad so I decided to move on.

Tekton

Tekton’s marketing was interesting so I took a look at it next. It seemed to be easier to set up and a couple random blog posts seemed to praise it so I downloaded another massive installation yaml which the guide told me to just kubectl apply -f which is okay I guess. I spent a few hours with it and it became clear you basically had to use it both for CI and CD so I put it to the side and moved on.

Flux

So why did I mention this one first but try it last? A few reasons. While researching, the v1 -> v2 development effort was going on and v2 didn’t have a lot of features but v1 was being phased out so I didn’t know if I should spend time learning v1 while v2 was coming along or if it’s worth jumping into alpha/beta stage software to avoid using legacy versions. By the time I did get around to it though v2 was clearly the way forward. Not only was it officially endorsed but among people that self-host, it was already the way to go 6-12 months ago. The docs were pretty clear, the cli was cross-platform and could be installed on a Raspberry Pi (something that argo at the time didn’t have available) and the examples worked. In an evening I had learned basically everything I needed to in order to just drop it in, generate the flux CRDs for my existing kubernetes manifests as well as helm charts and be up and running. The only issue I had, probably just a pet peeve of mine, was that the repo it creates on github is private by default and my goal is to have everything public and also secure (not there yet) but whatever. Flux v2 also mentioned as the gitops toolkit is made up of several different parts that have a specific purpose which is something I certainly appreciate. All in all, I think I’m going to stick with flux but you never know how homelab things will end up.

Conclusion

Flux is a really great project and I’ve enjoyed using it so far. The problem of continuous delivery has been solved for me and I recommend checking out if you’re in a similar position. Next up, I’m going to look at renovate bot for automating image updates, SOPS for secret management (which flux supports and an upgrade controller) to further help automate operations. Thank you for reading, I hope you enjoyed it and maybe learned something.

Homelab Current Form

Sun, 07 Feb 2021 14:35:32 +0200

The first part of the software changes I made was covered in the first part where I explained how and why I started going down this path.

Software choices

This whole journey was about making my home infrastructure better. Part of that was about having a way to more easily create the foundation on which kubernetes would run on, as well as describing my setup as code. Proxmox is a beast in its own right. Features include being able to manage kvm virtual machines and lxc containers, an API that can be used to interact with it programmatically and most importantly being open source. While researching what automation tools could interact with proxmox, I found a community-made terraform provider as well as a set of two ansible community modules (kvm and lxc). After thinking about it a little bit, I wanted to use terraform in a non-cloud environment but also gather some knowledge about it so that’s what I decided to use. I can have my provisioning requirements in a file that terraform understands and store it in git which is exactly what I was going for. Ansible is still in the picture since it is used to set up and configure the debian environment inside the virtual machines. Now that the hypervisor part is covered, let’s move on to running services.

Making it store is harder than making it run

As mentioned in the first part about software changes, I was learning kubernetes and now I could not only wipe and recreate the cluster itself but also the entire virtual machines that it was running on. Some time after starting to use proxmox with terraform I attempted to use rancher to manage kubernetes but ended up ditching it due to various problems with running even basic stuff on it (very likely that it’s a case of PEBCAK, I don’t think rancher is terrible or anything like that).

The problem

However the problem of shared storage continued to taunt me for the following months. There was seemingly nothing that a simple fella could do to use a nice and simple NAS running truenas core as storage that is able to be dynamically provisioned for use by kubernetes. Now, I hear you, “what about nfs-client-provisioner”, someone less familiar with this pile of madness might exclaim. Indeed it does exist and barely work except the helm chart for it is deprecated and it does not work with kubernetes version 1.20 and later since it does not seem to be using CSI drivers.

I really tried

Months of furious and frustrating testing ensued. Not only was I trying to get applications running on kubernetes, I was also fighting with the unexpectedly complex tast of using the storage server I had available.

glusterfs

I’ve gone through basically any and all commonly suggested options for dynamically provisioned storage. Prette early on I tried glusterfs with heketi by making 3 LXC containers and mounting an nfs share to each one that would serve as the brick in the gluster volume. Suffice it to say that it didn’t work and things were getting out of hand.

longhorn

After a bit more research and I found out that longhorn, another project by the authors of k3s, used iscsi to communicate with kubernetes. That could work I thought, except there was no coherent example of how to use it without using targetd for iscsi. Longhorn was starting to look more appealing, I could just put the virtual drives of the VMs on an nfs share, run longhorn inside the VMs to pool all their storage together and call it good enough. No, I could not give up yet. A sub-optimal solution would do if there was no other way but I was convinced something more was out there. My patience and hope were running out but not empty yet.

success

That all changed in early January where during my nearly daily search for possible storage solutions I hadn’t tried, I found out about democratic-csi. This was it. Made to be used with freenas, truenas as well as DIY ZFS setups. The silver bullet was here. Just a few minutes of reading about it and writing my configs, a short(read: long) helm command later and…success! The test pod was using the newly created storageclass that was backed by the nfs share on truenas. After that I rushed to deploy my standard set of gitea, droneci and minecraft to test it out and it was working for real without a hitch.

Conclusion

This was a long journey and it isn’t coming to a close any time soon. The next step is to continuous deployment but that’s an issue for another time. To conclude as briefly as possible, I’ve no put most of the critical pieces of my homelab in a single git repo that I can use to recreate almost all of it from scratch with minimal manual intervention (there are still a few quirks like not having dynamic inventory for ansible and having to manually copy IPs but I can put that aside for now). Despite the imperfections like not having dynamic inventory for ansible or having secrets stored unencrypted in git, I’m very happy with the setup is working. If you missed them, make sure to check out the hardware updates and the first part about software changes for a better of what I’ve been up to.

Homelab Evolved

Sun, 07 Feb 2021 13:53:10 +0200

The hardware side of the changes made to my homelab were covered in a previous post where I also alluded to some software changes.

What came before

This is the first part explaining more in-depth the issues that I had and how I dealt with them. Let’s set the stage first, my homelab was working fine in many ways. It was nice and stable debian install running kvm virtual machines using qemu and libvirt with virt-manager to perform the simpler start/stop tasks. There was really nothing wrong with it, I was exploring different software, running some services on the local network and even hosting a public website on it.

Good enough is not good enough

However as someone interested in infrastructure a few things were bothering me. Sure, I was using docker-compose with traefik for my public website and services. Yes, I was using ansible to manage the configuration of most virtual machines. But “good enough” and “I guess it works” doesn’t cut it. On the upside, I really liked having ZFS snapshots on a 2x2TB mirror. The ability to go back to a point in time where things were not broken had saved me a couple times when I inevitably broke one thing or deleted a file that could be replaced but would take a while. My backups were also on that ZFS mirror, on a different dataset of course, and that also very useful. I knew I wanted ZFS and that there was much room for improvement in regards to handling how I was running services. The single docker-compose virtual machine for the public stuff and another one for local stuff were mostly adequate but the manual management was not. Not to mention that and all my precious stuff was on one box which is well below ideal.

Taking the step

One day I decided it was time to move up in the world. Despite using docker and docker-compose for years I had never dived into kubernetes because it seemed difficult and complex. New syntax, new system with its own architecture and internal structure, new workflow, new everything. I decided to set up 3 VMs and played with default k8s as well as k3s on them. Dipping my toes into it was pretty fun and since nothing depended on it I was free to wipe it and start over. The same issue kept cropping up, having to manually install debian thrice and set up kubernetes just to delete all of it and redo it was again, less than ideal Initially I started writing a playbook before finding out that someone smarter had already gone down this path. Setting up k3s-ansible was easy so there was at least kubernetes setup automation but the whole thing was too thrown together there was no shared storage. However, I was becoming more familiar with kubernetes and learning the concepts as well as how to write yaml manifests.

Conclusion

At some point it became too bothersome, a lot of manual work was involved and basic stuff like copying over ssh keys was also done mostly manually so I had to move on. While all of this was going on, I had started planning what hardware updates I would get which you can read about here: Hardware Updates 2020

Hardware Updates 2020

Fri, 15 Jan 2021 21:10:38 +0200

Getting a real storage server

During 2020 I got a few upgrades to my setup but I don’t remember the exact order of them. The most exciting was a 4U rackmount case that I could use to put up to 6×3.5″ hard drives by default and up to 9 with a 2×5.25″ to 3×3.5″ hotswap bay. Also I bought 3x4TB ironwolf hard drives to put in it initially because my 2x2TB mirror was regularly reaching 95% capacity meaning I had to prune the snapshots regularly and couldn’t use it to store more files on it. This made me warm and fuzzy inside but the old mirror had to sit on the bench for this transition. The power supply in my server had only 4 sata connections which would be enough for a boot SSD and one of the arrays but not everything at once. Initially that wasn’t a problem, installed truenas core on the ssd, set up the 3 4TB hard drives in a raidz1 configuration and started creating datasets and shares.

Adding some extra spice

After a while I wanted to access the data on the old miror which meant having to buy a new power supply so I got one. Namely the Corsair CV550 which has 7 sata power connectors, more than enough to power the five hard drives and the one SSD. Moving on to issue number two, data connections of which the motherboard only had 4 of. Admittedly 50 euros for an extra 4 sata ports seemed excessive but I saved up and acquired a nice and simple pcie card that did the job, no raid controller of course. Alright, we’re cooking with gas now, storage is taken care of.

Dealing with separation of concerns

However, I used to have a multi-purpose server and now it is just a NAS so I can’t run all my services anymore. Fear not because the trusty old and used Dell Optiplex came to the rescue. A shop near my house had one with an i5 2400 and 4gb of ram as well as a 250gb hard drive with windows on it for just 130 euros so I jumped on it. First things first, I can’t be using a mechanical hard drive for the operating system and proxmox doesn’t dual boot so the HDD has to be replaced. Luckily the 240gb SSD from when I originally built my desktop is still alive and it’ll do more than fine for this purpose. The hard drive will be put aside for now but rest assured I do plan to use it at some point.

Serially accessing memories about ram upgrades

More importantly, I got a handle on the ram situation of my NAS. Upgraded that beast from 12 to 16 gigabytes of ram which left the 4gb stick as a spare. Not being one to miss out on some extra ram, the spare got added to the optiplex. Going from 4 to 8 is nice but I want to run more than 2 virtual machines. With that in mind and a deal available, I added an 8gb to the optiplex. So we’re up to 16(8+8) gigabytes for the NAS and 16(2+2+4+8) gigabytes for the server.. An odd configuration but ram is ram and you bet I’ll use it.

Conclusion

Perfect, we had storage and now we have compute as well. How was this hardware used though? Check back later (or read the next post if it’s up) to find out.