Home on inherent site

One Billion Rows Challenge

Mon, 01 Jun 2026 21:32:29 +0300

A couple of years ago I found an interesting programming challenge that includes reading a lot of data and calculating aggregates on them. It’s called the “One Billion Rows Challenge” or 1brc for short. I decided to take on the challenge to see how far I could get and I’d like to introduce others to it because I had fun participating.

The challenge

As the original challenge blog post and the equivalent github repository README both explain, is the following:

Your mission, should you decide to accept it, is deceptively simple: write a Java program for retrieving temperature measurement values from a text file and calculating the min, mean, and max temperature per weather station. There’s just one caveat: the file has 1,000,000,000 rows!

The text file has a simple structure with one measurement value per row:
1
2
3
4
5
6
Hamburg;12.0
Bulawayo;8.9
Palembang;38.8
St. John's;15.2
Cracow;12.6
...
The program should print out the min, mean, and max values per station, alphabetically ordered like so:
1
{Abha=5.0/18.0/27.4, Abidjan=15.7/26.0/34.1, Abéché=12.1/29.4/35.6, Accra=14.7/26.4/33.1, Addis Ababa=2.1/16.0/24.3, Adelaide=4.1/17.3/29.7, ...}
The goal of the 1BRC challenge is to create the fastest implementation for this task, and while doing so, explore the benefits of modern Java and find out how far you can push this platform. So grab all your (virtual) threads, reach out to the Vector API and SIMD, optimize your GC, leverage AOT compilation, or pull any other trick you can think of.

The rules (I’m quoting the github README here):

Any of these Java distributions may be used:

Any builds provided by SDKMan

Early access builds available on openjdk.net may be used (including EA builds for OpenJDK projects like Valhalla)

Builds on builds.shipilev.net If you want to use a build not available via these channels, reach out to discuss whether it can be considered.

No external library dependencies may be used

Implementations must be provided as a single source file

The computation must happen at application runtime, i.e. you cannot process the measurements file at build time (for instance, when using GraalVM) and just bake the result into the binary

Input value ranges are as follows:

Station name: non null UTF-8 string of min length 1 character and max length 100 bytes, containing neither ; nor \n characters. (i.e. this could be 100 one-byte characters, or 50 two-byte characters, etc.)

Temperature value: non null double between -99.9 (inclusive) and 99.9 (inclusive), always with one fractional digit

There is a maximum of 10,000 unique station names

Line endings in the file are \n characters on all platforms

Implementations must not rely on specifics of a given data set, e.g. any valid station name as per the constraints above and any data distribution (number of measurements per station) must be supported

The rounding of output values must be done using the semantics of IEEE 754 rounding-direction “roundTowardPositive”

This was written from the perspective of the author wanting more people to learn about features in Java versions after 8 and possible performance optimizations that can be made using those features. It also points to alternative compilers and runtimes so people can experiment more with parts of the software stack that might be considered infrastructure and as such neglected by developers. When I read about it, I liked the idea of the challenge. One billion rows is a significant amount, the file itself is 13GB on my system so doing anything with it in a reasonable amount of time is complicated. I mean, even just generating the input is complicated enough, let alone implementing something that reads a 13GB file and computes aggregates in the end. The rules provide details that inform which optimizations we are allowed to make and which ones are off the table.

The premise got me curious enough to sit down and write a program that did the naive thing knowing it was sub-optimal at best. I spent several evenings optimizing it, failing to do so and then trying again. Having seen the leaderboards (both the official ones for Java as well as community-built ones for other languages) I know my solution is not fast and there is room for improvement.

My proposed alternative challenge

I like the rules of the original and I would like to enhance them. Mainly, I’m not interested in a Java-only solution, I’d like everyone to be able to use whatever language they prefer. The restriction of not doing build-time calculations stands and I want to expand it, the measurements file should only be accessed at runtime so something like using Go’s embed.FS is not allowed. Other than that, I would suggest trying to avoid memory mapping the file and using unsafe parts of a language at first and see how far you can get without it. And with that, I think the challenge is explained well enough so we can get started with looking at my take on it.

Solution in Go

My solution for the challenge is located at gitlab.com/insanitywholesale/ongoing/x/1brc. It’s a single file and split into functions with plenty of comments so I don’t forget what I was trying to do which hopefully makes it easier to understand.

The `TempStat` struct

Starting off with the data structure that stores the aggregates for one station, we have the TempStat struct:

// TempStat stores measurement data for a station
type TempStat struct {
	Min   int64
	Max   int64
	Sum   int64
	Count int64
}

This is the final form after optimizations so there are a couple surprising things. First, the numbers are floating point in the input data but here we have integers. This is because it was faster to multiply by 10 (we know that values are in the [-99.9,99.9] range) because comparing integers is faster (but for parsing the opposite is true). Second, this is per-station so we need some place where all the results are stored, this is done in main() in a sync.Map which we will look at afterwards. We also don’t calculate the average directly, we’re storing the sum and count which allows for calculating it once at the end.

Opening the file

To start off, we need to open the file generated from the challenge repository. I haven’t done any optimization here although I’m aware others have memory-mapped the file. This is a simple function that returns a os.File:

// OpenMeasurementsFile reads and returns the file containing measurements
func OpenMeasurementsFile() *os.File {
	// Assume file is next to binary
	fileName := "measurements.txt"
	// Unless it's provided as a CLI argument
	if len(os.Args) > 1 {
		fileName = os.Args[1]
	}
	// After determining the path, open it
	file, err := os.Open(fileName)
	if err != nil {
		// If it doesn't exist just panic, there is no point continuing
		panic(err)
	}
	// And return the handle
	return file
}

It’s fairly obvious this is for a challenge, normally I wouldn’t leave a user-facing stacktrace from panic(). Here it worked well enough to remind me “you don’t have the file here and you didn’t provide it as an argument, plzfix kthx bai!~”.

Processing the input

Now that we’ve opened the file, we have to start reading it and doing our calculations. As a performance optimization, one that most people would do by default, I’m reading the measurement and updating the statistics all once. However, a less obvious one that I implemented was to read more than one line at a time. For this reason I have the ProcessLineGroup function which receives a list of lines as a first argument (through a channel, I’ll explain more in the part about the main() function) and then sends each one of them to the ProcessLine function for the actual work to be done. In this case, ProcessLineGroup() acts like a barrier between the goroutine-aware main() and the unaware ProcessLine(). The wg referenced here is a global variable defined as var wg sync.WaitGroup. In Go, a sync.WaitGroup is a construct used for synchronization. It has a counter which gets incremented by Add(x) and decremented by Done(). When the counter is zero, all goroutines being blocked by the waitgroup will then continue. It also takes a second argument, the results map, to make it accessible to ProcessLine. Here is the code definition for the above:

// ProcessLineGroup reads a batch of lines from the channel and processes them
func ProcessLineGroup(c chan []string, results *sync.Map) {
	// Mark goroutine as being done after ProcessLine exits
	defer wg.Done()

	// Read groups of lines from the channel
	for lines := range c {
		// Process each line in the group
		for _, line := range lines {
			ProcessLine(line, results)
		}
	}
}

Going a bit deeper, we’ll look at ProcessLine(). After a single line, one measurement about one station, has been passed to be processed, let’s see what happens to it. We know the format is station name then a semicolon and then a floating-point temperature reading. The fastest idiomatic way I found for doing this was to find the index of the semicolon and then make two string slices, one starting at the beginning of the line and ending before it and one starting after it and ending at the end of the line. Next, we parse it as a float because it’s faster (I played around with this part quite a bit, it’s been over a year since I wrote the program so I’m not sure how I was parsing it as an int) and also discard any errors to avoid an extra if. Then we get into the conversion that was hinted at by TempStat and store that value in the temporary temperature variable temp Since we have guarantees about the input we can afford to do this without checking, in any other case this would lead to incorrect results. The results.LoadOrStore call either returns the value for the key, station here, or it stores and returns the value we pass as the second argument which is a reference to an instance of TempStat with the min, max, sum set to the only measurement value we have for that station and count set to 1 since we’ve only seen it one time. The return value from the results sync.Map is any so we need to assert its type as a pointer to TempStat. The pointers and references are used so we don’t pass around the concrete instance of the struct which would be more data and therefore slower. At the end of the function there is a simple update of min, max, sum and count for the station. The if statements are separate and look a bit weird, it ended up being a bit faster to do it this way and also less duplicate code than the if..else if..else version. And with all that explained, here is the definition of the ProcessLine function:

// ProcessLine processes a single line and updates temperature stats in the map
func ProcessLine(line string, results *sync.Map) {
	// Locate the index of the delimiter,
	delimiterIndex := strings.Index(line, ";")
	// and split the line according to that.
	station, measurement := line[:delimiterIndex], line[delimiterIndex+1:]
	// To store the measurement, first read it as a float (faster than ParseInt),
	tempFloat, _ := strconv.ParseFloat(measurement, 64)
	// and then since we know it'll be in the [-99.9,99.9] range multiply by 10 and store as int
	temp := int64(tempFloat * 10)

	// Atomic map update using sync.Map
	value, _ := results.LoadOrStore(station, &TempStat{Min: temp, Max: temp, Sum: temp, Count: 1})
	// Get stats for relevant station
	stat := value.(*TempStat)

	// Update station temp stats
	if temp > stat.Max {
		stat.Max = temp
	}
	if temp < stat.Min {
		stat.Min = temp
	}
	stat.Sum += temp
	stat.Count++
}

Round and Print

Grouping the output and the rounding might seem a bit odd but the challenge mentions rounding only with regards to output. There is no other point in the code where rounding is relevant so I’ll explain both of these together. I decided to not invest too much time in this part so it is fairly idiomatic and can most likely be improved.

The Round function just calls out to math.Round which looks like it does the thing the challenge asks for. Being frank, I do not remember why the times 10 and then divide by 10 is in here. There is a special case for zero implemented, I think my result failed validation otherwise.

// Round does roundTowardsPositive
func Round(x float64) float64 {
	rounded := math.Round(x * 10)
	if rounded == -0.0 {
		return 0.0
	}
	return rounded / 10
}

The PrintOutput function is generally pretty boring outside of the sync.Map usage and the int to float conversions. The original input data has floating point measurements, we converted them to integers for processing speed but now we need to calculate the average and provide min/max/avg as floats. We pay the cost of calculating the average once per program run instead of once per line processed which is incredibly benficial when the dataset is one billion measurements. The printing works by getting the station names since that is the unique identifier here. There doesn’t seem to be a better way to do this so a function collects all the station names and stores them in a slice of strings and always returns true to satisfy the type of that argument (func (m *Map) Range(f func(key, value any) bool)). After that, the list of station names is sorted and we construct the output string. For each station, the integer values are converted to floats and reduced by an order of magnitude to normalize them to the original values and then passed through Round(). Last, we print the result to standard output:

// PrintOutput prints the final temperature statistics for each station
func PrintOutput(results *sync.Map) {
	stations := []string{}

	results.Range(func(key, value any) bool {
		stations = append(stations, key.(string))
		return true
	})
	sort.Strings(stations)

	finalResultString := "{"
	for idx, station := range stations {
		stat, _ := results.Load(station)
		tempStats := stat.(*TempStat)

		finalResultString += fmt.Sprintf(
			"%s=%.1f/%.1f/%.1f",
			station,
			Round(float64(tempStats.Min)/10.0),
			Round((float64(tempStats.Sum)/10.0)/float64(tempStats.Count)),
			Round(float64(tempStats.Max)/10.0),
		)

		// Add comma and space everywhere except for the last entry
		if idx != len(stations)-1 {
			finalResultString += ", "
		}
	}
	finalResultString += "}"

	fmt.Println(finalResultString)
}

The if for not adding a comma to the last entry bothers me and could probably be converted to use strings.Replace, probably something like strings.Replace(finalResultString, ", }", "}", 1) but oh well, this is how it is now.

Bringing it together in `main()`

Alright then, we’ve seen the individual puzzle pieces so far and connecting them seems somewhat intuitive but the code in main() was probably the thing I spent most of my time on. This is because there are several parameters that can be tweaked here, especially so because of the use of goroutines (A Tour of Go link). In the big picture, we’re looking to read a file with a billion rows and do a simple calculation as fast as possible. One of the obvious ways to make this fast is to split up the work. Easy to say but hard to do. How do we split the work? In how many parts? How big should those parts be? How many worker processes should we have? Answering all of those took a while, the Go profiler was a huge help here by pointing out where the bottleneck was. I believe I have done a decent job at explaining the code in the comments, take a look:

func main() {
	// Define amount of worker goroutines
	numWorkers := runtime.NumCPU()

	// Set GOMAXPROCS equal to amount of goroutines
	runtime.GOMAXPROCS(numWorkers)

	// Open the measurements file
	file := OpenMeasurementsFile()
	defer file.Close()

	// Create thread-safe map to store station statistics
	results := &sync.Map{}

	// Create a channel to send lines to workers
	lineChan := make(chan []string, 1000)

	// Set number of goroutines to wait for
	wg.Add(numWorkers)
	// Create goroutines based on predefined amount
	for i := 0; i < numWorkers; i++ {
		go ProcessLineGroup(lineChan, results)
	}

	// Set chunk size
	chunkSize := 64 * 1024
	// Create fixed-size buffer over which to send chunks
	buf := make([]byte, chunkSize)
	// Keep any partial line for later processing
	partialLine := ""

	for {
		// Read file into buffer
		n, _ := file.Read(buf)
		if n > 0 {
			// Split the buffer into lines
			chunk := partialLine + string(buf[:n])
			lines := strings.Split(chunk, "\n")

			// Save the last line if it is incomplete
			partialLine = lines[len(lines)-1]

			// Send lines to channel in batches
			if len(lines) > 0 {
				lineChan <- lines[:len(lines)-1]
			}
		} else {
			break
		}
	}

	// Send any remaining partial line
	if len(partialLine) > 0 {
		lineChan <- []string{partialLine}
	}

	// Close the channel after all lines have been processed
	close(lineChan)

	// Wait for all workers to finish
	wg.Wait()

	// Print the final results
	PrintOutput(results)
}

Now that you’ve seen the code, I have some notes in no particular order. The channel mechanism allows main() to send the lines to ProcessLineGroup() runnin in the multiple goroutines which then process the line groups created from a 64 * 1024 chunk of bytes. The goroutines are spawned by the go ProcessLineGroup(lineChan, results) call, prefixing a function call with go runs it in a goroutine. The channel and its buffer size is not pefectly tuned but 1000 seemed to be an okay spot to leave it at. The partial line workaround appears to be unnecessay, I think it’s a leftover.

And that’s my solution! Comparing it with ./calculate_average_baseline.sh > out_expected.txt shows that they are identical.

Benchmarks

I did a quick run using hyperfine, the same tool the original challenge author used. In order to avoid disk caching skewing the results I used the --prepare option to sync the filesystem and drop the disk cache:

cd ~/go/src/ongoing/x/1brc && go build
hyperfine --warmup 0 --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' './1brc ~/gitty/1brc/measurements.txt'

Time (mean ± σ):     42.469 s ±  0.654 s    [User: 118.403 s, System: 8.892 s]
Range (min … max):   41.712 s … 43.710 s    10 runs

So I’m at about 42-43 seconds. Not amazing but it’s at least under 1 minute.

How to get started

Looking at READMEs and figuring out scripts takes a couple of minutes so I think a short how to might help lower the barrier to entry. So here are the setup steps:

git clone https://github.com/gunnarmorling/1brc
./mvnw clean verify
./create_measurements.sh 1000000000
./calculate_average_baseline.sh > out_expected.txt

Now you have the measurements file and the expected output to compare to. You can save your solution’s output to a file (e.g. my_output.txt) and use something like diff --suppress-common-lines out_expected.txt my_output.txt to see if you got it right. After you’ve checked for correctness, it’s time to optimize and then re-check that you’re producing correct results but faster! You can use the hypefine command I used as well to check for speed.

Conclusion

My shell history says I started working on this challenge October 2024. I don’t know why almost 2 years later it dawned on me to write a blog post about it but I hope you decide to take it on and have as much fun as I did.

Home Assistant

Sat, 07 Jun 2025 14:12:34 +0300

I recently set up Home Assistant as a way of getting into home automation. It’s a really cool piece of software and took a bit of time to set up so I’d like to document everything I’ve changed here.

Getting started

Home Assistant is an open-source home automation solution, possibly the most well-known in its category. It’s a great DIY and self-hosted alternative to smart home offering from bigger vendors. As a platform it can integrate with tons of products available on the market and is very easy to run. I’ve been interested in using Home Assistant for a few years but I never had anything to use it for or with. Recently I decided I should look more into it so I ordered a SNZB-02D temperature and humidity sensor as well as a ZBdongle-E ZigBee stick also by Sonoff. ZigBee is a protocol that a lot of the smart home devices use to communicate, including my temperature/humidity monitor. This means that as long nothing goes wrong, I should be able to track my room’s temperature inside Home Assistant. Sonoff also has smart power plugs, look for the ones that have ZB in the model number like the S26R2ZB or S60ZBTPF (I didn’t check and wasted 10 euros).

I picked up the items I ordered, dusted off the old Raspberry Pi 3b+ I’ve had for years and installed Home Assistant OS on it. It took about an hour because for some reason the ethernet port on the Pi decided to not work. After facing its demons and working through the childhood neglect trauma I had given it, the ethernet blinky light blessed the little Pi. A little network setup and ta-da! I can log in using a browser on my desktop. Following that I shut it down, connected the Sonoff USB ZigBee stick and turned it back on. With a few clicks the USB stick was detected and ready to be used inside Home Assistant. I made it scan for devices and set the temperature and humidity monitor into pairing mode and then connected to it. This was a surprisingly straightforward process and right after there was data visible in the homepage.

A couple weeks later I picked up another two sensors and two zigbee power plugs. As a word of advice, the S26R2ZB is cheap so it only supports on/off functionality and not power measurement. The S60ZBTPF should support both though, I have the non-ZigBee version and it does but only using eWeLink.

Graphing the worst star’s waste heat

I keep the monitoring device on my desk so I can look at it and check that I’m not going crazy and the weather is indeed getting warmer. I also have one out in the balcony and one in the living room. After few days I could look at the pretty graphs and see how the temperature fluctuated throughout the days which was quite interesting. I could visibly see when I had opened the balcony door for some fresh air or how much the air-conditioner actually drops the temperature. In general the whole solution works very well. One day there was notification about a firmware updated for the ZigBee USB stick which I could install directly through Home Assistant. The sensors also had a firmware update and could also be updated directly through HomeAssistant. I don’t know why this seems like such a big deal to me but it’s surprising to see this process be supported and vendors not pulling some bs for once. I still hate warm weather and the sun but now I can track how much it’s affecting me.

Add-ons

For any web interface I consider it necessary to have an HTTPs certificate and some way to remotely access it. Thankfully using add-ons, it’s possible to get certs from LetsEncrypt, use nginx as a reverse proxy and also install cloudflared. Cloudflared is not official but it does work well even if it was a bit troublesome to set up initially. It does require some fiddling in the YAML configuration file (using the editor add-on) but it’s not too difficult. I also enabled prometheus metrics, which require authentication to scrape so it will be a slight bit more complicated than usual but nothing unusual. There is a lot of other software available as an add-on, many things which I don’t care to run on Home Assistant on my Raspberry Pi, but it’s good to see such robust support and that they are available for use cases where they make sense.

Later expansion

I plan to add more stuff in the future and start playing with automations. Currently it’s just a couple lights, the sensors and the power plugs. There is no inclination to do something extra with them, just having control of all of them through one place is helpful enough.

Conclusion

It’s a cool setup that works for what I want to do right now and also has a lot of room for expansion. Now that I can access it from the outside and have TLS certificates it been a very nice experience. If you’ve been waiting to start as well, it’s a lot easier than you might think.

Remote Access Homelab

Sun, 04 May 2025 18:26:25 +0300

One of the remaining things for my homelab to have is remote access. I set up a couple options and faced some issues during the process which I want to document here.

Tailscale

Tailscale is wireguard-based VPN solution. Its main idea is that you run it alongside whatever you want to be accessible in your tailscale virtual network and then you can get to it. My primary concern was replacing something like openvpn where it can be set up so it’s almost like being connected to the local network. In tailscale this can be achieved by making it announce that it can route to a specific subnet. This way things like local DNS works normally (for example I can access opnsense.home.inherently.xyzand see the web interface of my router). Additionally, you can make your traffic be routed through a specific device. This might be the router at home which will be marked as an exit node and then in the mobile app select it to be used as such. The choice of tailscale dates back to when I was still running pfSense. In pfSense there has been a built-in integration for quite a while that can work as an exit node and subnet router. In OPNSense it used to require a very manual install and setup process through the FreeBSD ports system but now there is a plugin which is a lot more user-friendly. I set it up that way and then joined my phone and my tablet to the tailnet which means I can access all of my servers remotely. Problem solved, let’s go home, roll the outro. Not yet. The thing is that while this works fine for just myself, if I want someone to check out some new selfhosted service I have or connect to a database I’m hosting I don’t want them to have to set up tailscale first.

Cloudflare Tunnel

To address that need, I chose Cloudflare Tunnel. This works a bit differently than Tailscale, you run cloudflared and tell it where traffic for a domain name should be redirected. In my case it runs inside kubernetes and points to the ingress controller. This… took a while for me to get working. It was mostly my own fault and lack of knowledge of how cloudflare works. There are certificates that cloudflare generates which encrypt traffic between clients and cloudflare. This process is done automatically and only for the root domain, for any subdomains you better get your wallet out. This meant that since I was trying to access services that were something like service.home.inherently.xyz, TLS was broken. Took about 2 days to figure this out and I only managed it thanks to Kryptonian in the Home Operationsdiscord server. The way I fixed the TLS problem was by just putting everything under the root domain which then made it work. There is a bit more work to do because I think cluster-template does have a good idea with separating services that should be externally accessible by running two ingress controllers. However, at this point it gets the job done so I’m reasonably happy with it.

Conclusion

Access do be external.

Talos Linux Setup and Configuration

Tue, 11 Mar 2025 19:28:45 +0200

After about two weeks since the first time I installed Talos, I think I’ve figured out a good setup. In the previous post I went over the PXE setup more than the kubernetes part of it so now I’ll cover the other half.

How the Talos configuration works

Following the getting started guide tells us that to generate the configuration files we need to run something like talosctl gen config homecluster https://10.0.50.69:6443. This command will generate 3 files. These are controlplane.yaml, worker.yaml and talosconfig. The control-plane nodes will use the first file, the worker nodes will use the second file and the third one should be placed into ~/.talos/config (with an edit to specify the node IP addresses or as Talos calls them in this case, endpoints). If we run the command we’ll get some output like this:

generating PKI and tokens
created controlplane.yaml
created worker.yaml
created talosconfig

Then, if we try to re-run the same command we will get a message like the following:

1
2

generating PKI and tokens
file "/home/angle/talos/controlplane.yaml" already exists, use --force to overwrite

If we do the “just go away and do it anyway” thing that it says and add --force it will again show the previous output. In this process, we’ve wiped all the secret tokens and certificates it generated before and cannot recover them. For a first-time installation this is barely an inconvenience, we probably do not care. When doing this for an existing cluster, we will also need to get a new kubeconfig since the secrets that are re-generated are also relevant for the certificate that is stored in the kubeconfig to access the cluster. That’s one of the reasons that if you’ve had the cluster running for a couple months, this might be a bit more annoying.

Separate secrets

The alternative is to take a more in-depth look at the documentation which will land us on a page with notes for production-level setups, specifically the section about separating out secrets. Following the examples in that page we can run something like talosctl gen secrets -o secrets.yaml first and then talosctl gen config --with-secrets secrets.yaml homecluster https://10.0.50.69:6443. This way, whenever we run the command to generate the configuration files (controlplane.yaml, worker.yaml, talosconfig) we will not also wipe the secrets. Now, why is that process useful or even preferable?

Using this approach, the secrets can be stored separately from the configuration itself, that’s the obvious part. The less obvious part is that given the same talosctl, the CLI tool, we can generate the same 3 files. Additionally, the kubeconfig that we can receive by running talosctl kubeconfig ~/.config/kube/config --nodes 10.0.50.72 --endpoints 10.0.50.72 is not going to need to be updated even if we reinstall Talos. You might then think, if we continually re-generate these 3 files, how can we ever permanently store configuration changes? Putting them directly in controlplane.yaml and worker.yaml is a no-go since they’ll be overwritten by a talosctl gen config command. That’s where patches come in.

Patches

The talosctl CLI tool has a couple helpful options for applying patches when generating configuration. Similar to having one configuration file for control-plane nodes and one for workers, the CLI flags are the same way. There are 3 flags: --config-patch-control-plane which applies the patch only to control-plane nodes, --config-patch-worker which applies the patch only to worker nodes and --config-patch which applies it to both. Using these, we can write patches and apply them as we wish to any and all of the nodes in our cluster. All 3 of the flags accept either inline JSON patches or YAML files depending on the notation. Now I’ll switch gears a bit and discuss my own specific setup as an example. I have 3 patches at the moment. First, is a baseline that configures network interfaces, defines the installation disk, sets up NTP, adds kernel parameters, enables workload scheduling on the control-plane nodes and certificate rotation for the kubelet. The other two are specific for the openebs storage system and for the metallb bare-metal loadbalancer. It’s important to note at this point that it merges the changes. So if you want to remove something from the default configuration (such as the node.kubernetes.io/exclude-from-external-load-balancers: "" node label) it’s not as easy as just having the key specified but being empty (for example nodeLabels: {}). To generate my configuration, I can run the following:

talosctl gen config homecluster https://10.0.50.69:6443 \
	--with-examples=false \
	--with-docs=false \
	--with-secrets=secrets.yaml \
	--config-patch=@patches/01_baseline-changes-patch.yaml \
	--config-patch=@patches/02_openebs-mayastor-patch.yaml \
	--config-patch=@patches/03_metallb-patch.yaml \
	--endpoints=10.0.50.71,10.0.50.72,10.0.50.73

Here I’ve disabled documentation and examples just so the file isn’t as cluttered. The important part though is that the sensitive information is only stored in only one file which could be encrypted and put into version control. From there, I could use the same version of the CLI tool to generate the final configuration file. The goal is to be able to have some sort of GitOPS setup where all or most of what’s needed to create the Talos configuration is stored in the same infra repository as everything else. One tool that seems to fit this exact use-case is talhelper but I’ve not looked into it yet.

Conclusion

I keep finding cool stuff about Talos and the homelab tinkering will continue until morale improves.

PXE setup for Talos Linux with Matchbox

Sat, 01 Mar 2025 19:00:33 +0200

Join my campaign of hate against installing operating systems interactively. I usually wipe my homelab virtualization hosts every week or two. This might sound insane but I’ve been working on automating their setup. As explained in the previous post, friendship ended with ProxMox and kubernetes will take its place. Specifically Talos Linux, a linux distro you can’t even SSH into. ProxMox kind of notoriously doesn’t have a way to be sanely installed through PXE but Talos is kind of made for it. However, PXE (and iPXE, we’ll discuss this later) is not perfectly documented and can be even harder to debug. So in this post I’ll document what worked for me and talk about some pretty helpful resources.

Plan

PXE

So let’s start with a bit of background, then lay out what I wanted to achieve, why and what I considered “good enough”. First off, PXE. It stands for pre-boot execution environment and is mostly reliant on motherboards and network cards to be implemented on the client side. It’s a stripped down environment before the computer is booted into an operating system. Inside it, there are standards for getting information about what to execute, where to get it and what configuration to use. PXE is the name of the general concept and the first version of the technology. It allows booting a computer from the network by loading files from a TFTP server. Later on, iPXE came around to fix some of its limitations. One thing it’s useful for having a computer and being able to boot it from the network and install an operating system on it without having to have a USB stick on you. With something like syslinux you can even create menus to be able to choose what to install and have a bunch of different options with different settings. As computer installations grow larger, it’s easy to see how this becomes kind of essential as you can’t run around multiple datacenters any time something needs to be re-imaged. A counter-point to that could be that servers have remote management features allowing us to load ISOs as if they’re DVDs but even then, iPXE has an advantage because you can fully automate the process. Also my poor version of servers (read: used office PCs) do not have IPMI. At any rate, that’s the general idea of PXE.

Talos

How does that apply to my case then? I want to be able to completely wipe all three systems and then effortlessly restore them. As I plan to run kubernetes on it, the installation of applications to the cluster is taken care of using FluxCD so I only really need to solve the installation part. That’s where Talos comes in. Talos is an incredibly minimal linux distribution that is designed with the sole purpose of running kubernetes. No SSH, no TTY, no nothing. Sitting in front of the system only allows you to change kernel boot parameters and the network configuration. Everyhing is done using the API which is accessible interactively through the talosctl CLI utility. This means that we can have our operating system and kubernetes configuration as code for the price of one config file. Sign me up!

Matchbox

Those are the ingredients but it can hardly be called a meal. We need to get some kitchen utensils in this analogy to get the desired result. Our PXE setup will need to have a little bit of logic in it so it knows what to install with what configuration. See, Talos can be told to fetch a configuration file from an HTTP server using a kernel parameter. A conditionally-rendered file would allow us to alter the response depending on who’s requesting it. This is where matchbox slots in. When matchbox gets a request it can see some information about the host requesting it and therefore make a determination about what to reply with. We can then serve different options according to which system is asking such as what Talos configuration file to use. This way the control-plane kubernetes nodes can receive the Talos controlplane.yaml while worker nodes get the worker.yaml. So a brand-new host with an empty disk drives would boot up, skip the disks since they don’t have an operating system and then try booting using PXE. Then the host would get its configuration, install the operating system, install kubernetes and become part of a cluster.

Definition of Done

The ideal process when getting adding a host should look like:

note down the MAC address and corresponding IP somewhere
plug in the cables
configure the boot device order
save settings and exit
play minecraft while waiting
run one or zero commands
host shows up in kubectl get nodes
recreating it from scratch should be just as easy

I don’t know if I can get exactly to that point but I’m willing to put in some effort into the setup so in the future I can avoid it to a greater extent. Some compromise is acceptable since I don’t have any IPAM infrastructure in place already so this isn’t the full and proper way to go about it.

Implementation

How do I do X in Y minutes

Meat and potatoes first, for quick reference here is what I did (5-6 hours of suffering omitted for brevity):

Installed tftp-hpa on Arch
Downloaded undionly.kpxe and ipxe.efi
Put them in /srv/tftp

Created matchbox.ipxe with the following contents:

1
2

#!ipxe
chain http://10.0.20.50:8080/boot.ipxe

Installed matchbox using Docker by running the following command:

`1`	`docker run --net=host --rm -v /var/lib/matchbox:/var/lib/matchbox:Z -v /etc/matchbox:/etc/matchbox:Z,ro quay.io/poseidon/matchbox:v0.10.0 -address=0.0.0.0:8080 -log-level=debug`

Downloaded vmlinuz-amd64 and initramfs-amd64.xz
Put them in /var/lib/matchbox/assets
Download talosctl
Run talosctl gen config homecluster https://10.0.50.69
Edit controlplane.yaml until it looks good
Put controlplane.yaml in /var/lib/matchbox/assets as well

Create /var/lib/matchbox/profiles/control-plane.json:

{
  "id": "control-plane",
  "name": "control-plane",
  "boot": {
    "kernel": "/assets/vmlinuz",
    "initrd": ["/assets/initramfs.xz"],
    "args": [
      "initrd=initramfs.xz",
      "init_on_alloc=1",
      "slab_nomerge",
      "pti=on",
      "console=tty0",
      "printk.devkmsg=on",
      "talos.platform=metal",
      "talos.config=http://10.0.20.50:8080/assets/controlplane.yaml"
    ]
  }
}

Create /var/lib/matchbox/groups/control-plane.json:

{
  "id": "kube01",
  "name": "kube01",
  "profile": "control-plane",
  "selector": {
    "mac": "18:03:73:2a:d8:a2"
  }
}

In OPNSense (link to my router)
- Under DHCP Static Mappings for this interface. add the hosts
- Under TFTP server:
  - Set TFTP hostname: 10.0.20.50
  - Set Bootfile: undionly.kpxe
- Under Network booting
  - Set next-server IP: 10.0.20.50
  - Set default bios filename: undionly.kpxe
  - Set iPXE boot filename: matchbox.ipxe
Boot Optiplex
Wait
Watch text scroll on screen
???
No profit, I spent my whole weekend on this

Explanation

PXE is an under-documented part of the common network services one can set up. An amusing yet educational thing I found was a redditor’s 3-part descent into madness and consequently enlightenment:

The matchbox documentation was also very good at explaining both the general process and giving some specific recommendations:

With those out of the way I’ll make an attemt at explaining it myself. When a PXE client starts, it does the DHCP dance which tells it where the TFTP server is and what the name of the file it should execute is. That file is a network boot program which can optionally load some configuration (also stored on the TFTP server). So after all of that is figured out, the PXE client will pull the kernel and initramfs, again from the TFTP server. In contrast, iPXE uses scripts stored on the TFTP server instead of plain configuration files so we can do fancier things with it. However, my old desktop PCs do not have iPXE so I had to use an in-between thingy, undionly.kpxe. What that does is load iPXE firmware (ipxe.efi) so we can access the cool new features without having to replace any hardware. When that loads, it talks to the DHCP server as an iPXE client this time and gets told to grab the matchbox.ipxe file from the TFTP server. That file is an iPXE script which tells the client to check the matchbox server. When matchbox gets the request, it checks the groups it has registered and if there are any matches according to the selector field, it uses the corresponding profile. In this case, the profile is for control-plane Talos Linux nodes. The kernel and initramfs listed in the profile are used alongside some specific kernel parameters. The most important parameter is talos.config=http://10.0.20.50:8080/assets/controlplane.yaml which tells the machine that is booting the operating system to fetch the configuration file listed there. Of course, you will need to have generated the configuration (using talosctl gen config homecluster https://10.0.50.69:6443) and moved it in the matchbox assets directory (or in another HTTP server, matchbox is just convenient in this setup). That configuration has some basic things like which disk and network settings to use. When the machine gets this configuration, it installs Talos and then waits. At this point the system is booted normally, no PXE involved.

Following that, all we need to do is run talosctl bootstrap --nodes 10.0.50.71 --endpoints 10.0.50.71 and we get a kubernetes cluster. To get the kubeconfig required to access the cluster, run talosctl kubeconfig my-talos-kubeconfig --nodes 10.0.50.71 --endpoints 10.0.50.71 and then check that it works by running kubectl --kubeconfig my-talos-kubeconfig get nodes. I’ve tested the whole process end-to-end twice now to ensure that nodes with blank disks and correct boot order will behave correctly and it’s almost magic. True enough, after pressing the power button on all three systems, making some coffee, and then running the commands mentioned previously, I got this output:

$ kubectl get nodes -o wide
NAME     STATUS   ROLES           AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION   CONTAINER-RUNTIME
kube01   Ready    control-plane   3h20m   v1.32.2   10.0.50.71    <none>        Talos (v1.9.4)   6.12.13-talos    containerd://2.0.2
kube02   Ready    control-plane   3h20m   v1.32.2   10.0.50.72    <none>        Talos (v1.9.4)   6.12.13-talos    containerd://2.0.2
kube03   Ready    control-plane   3h20m   v1.32.2   10.0.50.73    <none>        Talos (v1.9.4)   6.12.13-talos    containerd://2.0.2

And I guess that’s it. It needed some effort and research and the approach should be adjusted according to the network situation (the matchbox docs saved the day here). In the end though, I really like what I got working and it was a fun way to spend the weekend. I can go from basically brand new computer with nothing on it to full cluster with a high availability control-plane using patience and a single command. Two commands and I can install fluxcd on the cluster which will install all my applications. Looking back at the definition of done I set in the beginning, I consider this a success.

Conclusion

As the proud owner of a Talos Linux kubernetes cluster installed in a fully automated way, I am quite happy. I can get back into running services at home and live through the pain of using kubernetes daily again. I have a couple of ideas in mind for how to use my new container running platform but that’s for another post. Hopefully you got a better idea of what PXE is, how it works and how I used it in this instance. If you weren’t familiar with any of this, maybe it’s time to get your hands dirty and prepare to throw away your USB drives. Either way, that’s all for now.

Homelab 2025 Plans

Fri, 28 Feb 2025 22:51:33 +0200

Intro

After recapping the state of my homelab and writing down how it was in the end of 2024, I think it’s time to look towards the future. Having finally finished my thesis, I now have less constant stress to deal with and a bit more time than I’ve had in the last 2 and a half years. The main goal of rebuilding my homelab is to make it an actual production environment. This means that a few important problems need to be solved before it’s considered done. Along with that, I’m willing to lose a bit of flexibility in exchange for having something that actually works which is a bit odd for me to write but alas.

Hardware

Over the past several years I’ve invested quite a bit into the hardware that already exists. I won’t go into great depth about it, all the details are in the previous post. My homelab is housed in a 27U rack, includes a managed switch, a DIY router, 3 virtualization hosts and at least one NAS. That is more than enough to at least get started running several services. I will try to avoid investing in hardware this year with two exceptions, a VGA KVM and a good NIC for the router. Other than those two additions, I plan to focus on software choice and configuration.

Software

Kubernetes

I’m going bare metal kubernetes. This might seem counter-intuitive but it’s actually the easiest thing to get working. I still like ProxMox, I’m happy with the terraform provider for working with ProxMox and there is now a healthy alternative one, I use ansible at work and it’s all nice but I’m looking for an even higher level of automation. The integrations available for kubernetes are at a great point and have been for a couple of years now. I can forego dealing with issues such as “how do I make DNS automatically update when I start running a new service” and focus more on running and using my self-hosted services. Of course, kubernetes presents its own set of challenges and has its own quirks but I believe the tradeoff to be worth it for me.

The plan right now is to go with Talos Linux so my hosts are nice and lean. The entire process of installation can be automated using the matchbox PXE server and a bit of configuration and the operating system has just enough to run kubernetes and that’s it. I could of course regret this in which case I’ll go back to Debian my beloved but it remains to be seen. Initially I’ll go with a simple ISO installation to get started and then I’ll see if I want to take it a step further. The kubernetes-related stuff is mostly figured out, default CNI, nginx for ingress, metallb for external IPs, flux to deploy stuff and I’ll try adding renovate bot to help with maintenance. What remains is to decide if I’m going to go with openebs or longhorn for storage but that shouldn’t be too difficult or time-consuming.

Router

The other interesting software choice is the router. Having a dedicated machine for it seems wasteful and I’m bound to running whatever software is available for FreeBSD. No, I’m not planning on moving away from OPNSense at the moment but I do want to start the vicious cycle again. The plan is to have it in a VM, thus the vicious cycle, and then be able to run things like matchbox on the same machine, a different NTP server, a different DHCP or PXE server (mainly tinkerbell comes to mind), different DNS server (Hickory seems quite interesting) and so on. Those could be containers inside other VMs or whatever else, I’m not bound to whatever is packaged for FreeBSD or even specifically for OPNSense. This of course requires a hypervisor (ProxMox makes a comeback) and a NIC that can be passed through to a virtual machine in order to run the router. This is a secondary concern since I do have an old machine from the core2duo days if I just want to runs some basic network infrastructure like the ones mentioned previously for experimentation purposes. VyOS has also been on my mind but that’s a bigger jump and would require quite a bit of effort to adapt everything I have to it. The router plans are unclear at the moment so it’s more of a backburner project.

Conclusion

One must imagine Sisyphus, he rolls the rock and I rebuild my homelab. Was there a point to writing this? I’m not sure. These are my plans at the moment so I can get back into the hobby. If I do go through with them, I’ll try to document the experience when I have something workable.

State of Homelab 2024

Mon, 27 Jan 2025 22:38:48 +0200

When operating a homelab there is always something new to do or something waiting to be fixed. This post’s aim is to detail the state of my homelab at the end of 2024 and beginning of 2025.

Router

The previous post was about virtualizing pfSense inside ProxMox but this setup is no longer in use. In its place, there is a new system running OPNSense on bare-metal. This was done for a few reasons, the main one being that I wanted to move away from pfSense. More than that, there was a circular dependency where the ProxMox host had to be up in order for the router to function but in order for the cluster nodes to talk to each other, the router needed to be online. In practice this was not a huge issue but it bothered me since it made experimenting more difficult because recreating the cluster meant taking down the router as well. I’ve lost a few things, mainly pfblocker-ng and native tailscale integration but they’re not as big of a loss as I had anticipated. In the end I think it was worth it, over the course of the past few years the company making pfSense has acted against the best interest of pfSense users and I did not feel like using their software anymore.

Virtualization

I’m still running ProxMox, although now it is a 3-node cluster. I’ve spent a pretty significant amount of time tweaking the bootstrap process from a fresh install to having the entire homelab operational. Somewhere along the line I tried making it so non-cluster operation of multiple hosts was supported but gave up on that eventually. Each of the nodes has a 250GB SSD used as a boot drive and for node-local storage along with a 2TB HDD used as a Ceph OSD. Since the last post about using ansible and terraform with ProxMox, I’ve figured out the process of adding tools to the debian image in the virtual machine template. There is also the option to deploy using GitLab CI so just making committing and pushing to the repo is enough to create a new virtual machine. The k3s virtual machine terraform configuration was improved significantly but I’ve not adopted modules yet which seems to be the next step for organization-related improvements. I also run all software directly on top of debian virtual machines without containers to practice writing ansible. At work, our team runs everything in Docker so it could be a good idea to start moving to something similar at home for the sake of practice.

Kubernetes

The kubernetes cluster got a significant update. I’ve adopted FluxCD to a greater degree, figured out secrets storage and a pretty good structure for the YAML files. The k3s cluster has 3 control plane nodes and 3 worker nodes, one per virtualization host. I’m using longhorn for storage with plans to try and use the Ceph cluster as an additional storage class. The networking setup consists of MetalLB to have an IP in the subnet and then run ingress-nginx as the ingress controller on that IP. No plans for checking out Gateway API have been made at current but it is a good and necessary step forward in the ecosystem, I plan to move to it after things are a bit less in flux. The DNS is manual right now due to the low amount of services but I’ve tested external-dns with PiHole and I’m aware of the OPNSense webhook option, it remains to be seen what I’ll end up using. The setup needs a bit more tender love and care but it’s running well and doing what I need it to. I have plans to move more stuff to the cluster after the foundation is solid and not just have my own demo applications in there. The thought has crossed my mind to completely do away with ProxMox and then run Talos Linux bare metal on the systems but I’d be losing out on some flexibility which I don’t want to commit to right now. All in all, kubernetes is still pretty fun and the 1.30 and later releases have made good quality of life improvements.

NAS

Since I upgraded my desktop, the old one’s internals were transplanted to a different case are now a second NAS. I’m saving up to get a big external drive so I can have an extra copy of the current main NAS so I can freely mess with them to have a solid backup solution. The changes in TrueNAS Scale regarding applications has left me waiting for things to settle down before I commit to a final setup. Overall nothing exciting on this front.

Hardware

I run mostly used office desktops and a basic managed switch. I have 3 Dell Optiplex machines: 2nd gen, 3rd gen and 4th gen Intel i5 CPUs with RAM sizes ranging from 16 to 32GB each. For storage they have a 250GB SSD and a 2TB HDD as mentioned earlier. The router is a Fujitsu equivalent desktop with a 4th gen Intel Pentium and 8GB of RAM as well as a 500GB SSD. My rackmount NAS is an i5 4570 with 16GB of RAM and 4x4TB HDDs in a 3-drive ZFS RAID-Z1 plus one hot spare. My backup NAS is an i7 4770 with 32GB of RAM and 2x8TB HDDs in a 2-way ZFS mirror. The main network switch is a Netgear GS724TV4, just a layer 2 managed gigabit switch.

Conclusion

I wanted to write all of this down to have a reference for myself when I inevitably change something and it might help someone else as well. Also the blog is back(?) after more than a 2 year silence, hopefully I will not neglect it for that long again.

Virtualized pfSense on ProxMox

Tue, 20 Sep 2022 01:12:49 +0300

Recently I went down the path of running my router inside a virtual machine. This is far from the first time I’ve tried it but it is the first time I have a solid long-term plan. Keep reading if you’re interested in doing something similar, this should be fun.

Background

Last time I ran my router inside of a virtual machine was in 2017 and it was documented in this article. The reason, at least as far as I can tell, that it didn’t stick is because I ran it on my desktop. Since then, I started my homelab where I can run software on machines that are online most of the time. A small Dell Optiplex is more than enough for my networking needs at this time and I have two of them. I picked the weaker one since it isn’t as useful for quickly spinning up and down VMs due to the old CPU. My plan is to run some LXC containers and KVM virtual machines alongside pfSense which is why I’m virtualizing it.

Preparation

The ProxMox installation on the weak Optiplex is configured the same as the main one for testing purposes. When using the system as a staging environment, I installed it using ZFS. This was a great choice for having a 1-to-1 copy of the production version but it uses a little bit more RAM than I’m willing to spare. Due to unfortunate circumstances it’s equipped with only 20GB of memory and using more than 2gb of that for the host limits what other things can run at the same time on it. With that in mind I chose to reinstall using the default LVM scheme. After the installation I followed the official PCI passthrough guide and rebooted.

PfSense VM Setup

Since I have 2 identical Realtek NICs but different revisions, I looked through the output of lspci -vvvv to identify the correct one. With the device ID noted, I created a q35 UEFI-enabled virtual machine and made sure to uncheck the Pre-Enroll keys option (that one took a while to figure out). Don’t forget to set the option Start at boot to make it operate as if it was a bare-metal installation. The installation process is very streamlined so I was up and running very quickly. I had to do a little dance of plugging/unplugging cables to my switch and assign a spare VLAN to a port for setup purposes but once that was done I uploaded the backup config. Reassigning interfaces and VLANs was a bit tedious however the whole thing didn’t take more than 20-30 minutes.

Results

The performance isn’t any worse and reliability is somehow increased despite the old box using an Intel NIC and the new one using Realtek. As far as consolidation goes, this is a huge increase and I can now run multiple other things on a single machine without relying on pfSense packages.

New Gentoo

Tue, 23 Aug 2022 15:09:04 +0300

As the summer is coming to a close and before the exam period starts I decided to reinstall gentoo on my university laptop. I don’t plan on doing an ordinary follow the handbook and whabam you’re up and running thing so I wanted to document it.

WARNING: this is an incomplete draft but I wanted to publish it just so it exists as a resource.

Background

The laptop I use for university has pretty lackluster specs which is to be expected considering I bought it for 180 euros about 5 years ago. Nevertheless, it does its job very well and I wouldn’t be surprised if I’ve written more code on it than I have written on my desktop at home. However I have a habit of putting different distros on it, sometimes wiping it even in the middle of a semester. It’s pretty critical since it is used for a few classes and because I tend to work on personal projects fairly often while at university but that doesn’t mean it gets the white glove treatment.

But enough about the laptop, we’re here to talk about gentoo. Generally I can get by on any distro but I’m not going to settle for less than I can have if I can swing it. There are several trends in the linux world that I dislike and the only distro that allows me to avoid most if not all of them is gentoo. At the time of writing I’ve been using it for over 4 years with no plans in the forseeable future to move to something else. While I do love it, not everything is perfect so we’ll need to add at least one overlay and maybe write an ebuild or two but I’m getting ahead of myself. What’s the plan?

Plan

There are a few main requirements that I’d like the system to meet (in no particular order):

no systemd
no udev or eudev
no logind or elogind
no consolekit
no usrmerge
no dbus
no pam
no polkit
no networkmanager
no pulseaudio
no multilib
no wayland
no lvm
no btrfs
no swap partition
no grub
no silly network interface naming
least amount of packages written in perl, lua, ruby and python
no packages in unnecessary programming languages (fortran, ocaml, haskell, racket and a few others that some packages pull in)

and probably some that I forget. It might not be possible to achieve all of them but I want to get as close as possible. There are also a couple things not on the list that I’d like to have eventually but won’t be doing this time, namely using ZFS instead of XFS and having a stripped down kernel. One might think that not installing certain software would be as easy as just not adding them, and I really wish it was. However as you’ll see later on some packages have hard (meaning non-optional) dependencies on some of the above even where it isn’t necessary which complicates things. With the plan laid out, let’s begin.

Installation media

I headed over to the gentoo downloads page, picked up the latest (at the time of writing) minimal installation ISO, which is install-amd64-minimal-20220821T170533Z.iso, and used dd to write it to a 4gb usb stick After booting into it in cached mode (this copies the contents of the ISO to RAM) I selected the default keyboard layout, set a root password with passwd, started the SSH service with rc-service sshd start and looked up the IP address using ip a. From my desktop I opened an SSH session ssh root@10.0.20.49 and started up tmux to make sure that even if I disconnected things would still keep running on the laptop and I could come back to it.

Partitioning and formatting the drive

When following the handbook, the first step after getting the network up is to partition our disk. In my case this is a cheap 240gb SSD I added to the laptop since the 32gb of on-board eMMC was not enough. There will be just two partitions, one that will be the boot and ESP and one that will be root. The handbook recommends making the boot partition 256MB but I don’t want to fill it up quickly if I’m experimenting with kernel configuration so I’ll make it 512MB. The root partition will take up the rest of the space on the SSD. When I reinstall with ZFS as the root filesystem I’ll probably put the boot/ESP partition on the eMMC so I can manage the entire SSD using only ZFS. However right now I’ll just format the boot one as fat32 and for root I’ll go with XFS. Next up we mount just the root partition at /mnt/gentoo, go into it and we’re ready to grab our stage3 tarball.

Stage3

Heading back to the downloads page I decided to go for the no multilib openrc tarball stage3-amd64-nomultilib-openrc-20220821T170533Z.tar.xz which is the latest one at the time of writing. I’m used to curl instead of wget so a little curl -O incantation later and the stage3 is ready to be unpacked with tar xpvf stage3-*.tar.xz --xattrs-include='*.*' --numeric-owner.

Minor configuration

I’m not going to do too much configuration before the system is independently bootable but we’ll take care of some basics. Sadly the installation media doesn’t have vim, only vi but I can make do.

make.conf

MAKEOPTS

First, I added MAKEOPTS="-j2" to better take advantage of the processing power.

USE

The big one, right. Initially I’ll just set the basics:

`1`	`USE="-systemd -udev -logind -consolekit -policykit -dbus -pam -networkmanager -pulseaudio -wayland -lvm -btrfs -perl -lua -ruby -python -fortran -ocaml -haskell -racket"`

repos.conf

Git syncing is generally faster and doesn’t seem to have a per-day sync limit so I want to use that over rsync however the stage3 doesn’t include git so we leave those options commented out for now.

[DEFAULT]
main-repo = gentoo

[gentoo]
location = /var/db/repos/gentoo
#sync-type = git
sync-type = rsync
#sync-uri = https://anongit.gentoo.org/git/repo/sync/gentoo.git
sync-uri = rsync://rsync.gentoo.org/gentoo-portage
auto-sync = yes
sync-rsync-verify-jobs = 1
sync-rsync-verify-metamanifest = yes
sync-rsync-verify-max-age = 24
sync-openpgp-key-path = /usr/share/openpgp-keys/gentoo-release.asc
sync-openpgp-keyserver = hkps://keys.gentoo.org
sync-openpgp-key-refresh-retry-count = 40
sync-openpgp-key-refresh-retry-overall-timeout = 1200
sync-openpgp-key-refresh-retry-delay-exp-base = 2
sync-openpgp-key-refresh-retry-delay-max = 60
sync-openpgp-key-refresh-retry-delay-mult = 4
sync-webrsync-verify-signature = yes

Pseudo-filesystem copy-pasta

Plopping this here for the future

mount --types proc /proc /mnt/gentoo/proc
mount --rbind /sys /mnt/gentoo/sys
mount --make-rslave /mnt/gentoo/sys
mount --rbind /dev /mnt/gentoo/dev
mount --make-rslave /mnt/gentoo/dev
mount --bind /run /mnt/gentoo/run
mount --make-slave /mnt/gentoo/run

And that about does it for the pre-chroot stuff.

Chroot

Now we change root into what is going to be our system, mount our boot partition and run the initial sync using emerge-webrsync. Next up git gets installed so we can switch to git syncing in the repos.conf file mentioned previously.

Tangent: (neo)vi(m)

I wanted to edit the root user’s .bash_profile to automatically include /etc/profile and change the prompt and very quickly found out that there isn’t even vi inside the stage3. There is the option to install it but I vaguely remember it pulling in some perl or ruby dependencies that I would rather avoid. Good opportunity to see if neovim does better in this regard. Here is a pretend emerge with USE flags I would set:

USE="X -perl" emerge -pv vim

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N     ] app-crypt/libmd-1.0.4::gentoo  259 KiB
[ebuild  N     ] x11-base/xorg-proto-2022.1::gentoo  USE="-test" 1082 KiB
[ebuild  N     ] dev-libs/libbsd-0.11.6::gentoo  USE="-static-libs -verify-sig" 407 KiB
[ebuild  N     ] app-eselect/eselect-vi-1.2::gentoo  3 KiB
[ebuild  N     ] dev-libs/libsodium-1.0.18_p20210617:0/23::gentoo  USE="asm urandom -minimal -static-libs -verify-sig" CPU_FLAGS_X86="-aes -sse4_1" 1812 KiB
[ebuild  N     ] x11-libs/xtrans-1.4.0::gentoo  USE="-doc" 182 KiB
[ebuild  N     ] x11-libs/libXau-1.0.9-r1::gentoo  USE="-doc" 316 KiB
[ebuild  N     ] x11-libs/libXdmcp-1.1.3-r1::gentoo  USE="-doc" 325 KiB
[ebuild  N     ] x11-base/xcb-proto-1.15.2::gentoo  PYTHON_TARGETS="python3_10 -python3_8 -python3_9 (-python3_11)" 145 KiB
[ebuild  N     ] x11-libs/libICE-1.0.10-r1::gentoo  USE="ipv6" 384 KiB
[ebuild  N     ] x11-libs/libxcb-1.15-r1:0/1.12::gentoo  USE="xkb -doc (-selinux) -test" 437 KiB
[ebuild  N     ] x11-libs/libSM-1.2.3-r1::gentoo  USE="ipv6 uuid -doc" 355 KiB
[ebuild  N     ] x11-misc/compose-tables-1.8.1::gentoo  1776 KiB
[ebuild  N     ] x11-libs/libX11-1.8.1::gentoo  USE="-doc -test" 0 KiB
[ebuild  N     ] x11-libs/libXt-1.2.1::gentoo  USE="-doc -test" 767 KiB
[ebuild  N     ] app-editors/vim-core-9.0.0099::gentoo  USE="acl nls -minimal" 16324 KiB
[ebuild  N     ] app-editors/vim-9.0.0099::gentoo  USE="X acl crypt nls -cscope -debug -gpm -lua -minimal -perl -python -racket -ruby (-selinux) -sound -tcl -terminal -vim-pager" LUA_SINGLE_TARGET="lua5-1 -lua5-3 -lua5-4 -luajit" PYTHON_SINGLE_TARGET="python3_10 -python3_8 -python3_9 (-python3_11)" 16321 KiB
[ebuild  N     ] app-vim/gentoo-syntax-2::gentoo  USE="-ignore-glep31" 20 KiB

Total: 18 packages (18 new), Size of downloads: 40906 KiB

Not too bad, let’s see if neovim can do better:

USE="-tui -nvimpager" emerge -pv neovim

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N     ] dev-lang/luajit-2.1.0_beta3_p20220127-r2:2/2.1.0_beta3_p20220127::gentoo  USE="-lua52compat -static-libs" 1048 KiB
[ebuild  N     ] dev-libs/tree-sitter-0.20.6::gentoo  2857 KiB
[ebuild  N     ] app-eselect/eselect-vi-1.2::gentoo  3 KiB
[ebuild  N     ] app-crypt/rhash-1.4.2::gentoo  USE="nls ssl -debug -static-libs" 408 KiB
[ebuild  N     ] dev-libs/jsoncpp-1.9.5:0/25::gentoo  USE="-doc -test" 211 KiB
[ebuild  N     ] dev-libs/libmpack-1.0.5-r3::gentoo  33 KiB
[ebuild  N     ] app-eselect/eselect-lua-4-r1::gentoo  0 KiB
[ebuild  N     ] dev-libs/libuv-1.44.1:0/1::gentoo  1272 KiB
[ebuild  N     ] dev-lang/lua-5.1.5-r106:5.1::gentoo  USE="deprecated readline" 217 KiB
[ebuild  N     ] dev-libs/libvterm-0.1.4::gentoo  68 KiB
[ebuild  N     ] app-arch/libarchive-3.6.1:0/13::gentoo  USE="acl bzip2 e2fsprogs iconv lzma xattr -blake2 -expat -lz4 -lzo -nettle -static-libs -verify-sig -zstd" 7258 KiB
[ebuild  N     ] dev-util/cmake-3.22.4::gentoo  USE="ncurses -doc -emacs -qt5 -test" 9553 KiB
[ebuild  N     ] dev-lua/lpeg-1.0.2-r101::gentoo  USE="-debug -doc -test" LUA_TARGETS="lua5-1 luajit -lua5-3 -lua5-4" 71 KiB
[ebuild  N     ] dev-lua/mpack-1.0.9-r1::gentoo  USE="-test" LUA_TARGETS="lua5-1 luajit -lua5-3 -lua5-4" 16 KiB
[ebuild  N     ] dev-lua/luv-1.43.0.0::gentoo  USE="-test" LUA_SINGLE_TARGET="luajit -lua5-1 -lua5-3 -lua5-4" 172 KiB
[ebuild  N     ] dev-libs/msgpack-3.3.0:0/2::gentoo  USE="cxx -boost -doc -examples -static-libs -test" 497 KiB
[ebuild  N     ] app-editors/neovim-0.7.2::gentoo  USE="lto -nvimpager -test -tui" LUA_SINGLE_TARGET="luajit -lua5-1" 10678 KiB

Total: 17 packages (17 new), Size of downloads: 34352 KiB

That’s too much lua so we’re going with vim instead but it was worth checking out. A little echo 'app-editors/vim X -lua -perl -python -racket -ruby -terminal -vim-pager' > /etc/portage/package.use/vim later and we’re set.

First encounter with udev

After syncing, I saw the eselect news read prompt and decided to take a look. I remembered that unfortunately the default now is to use systemd udev instead of eudev so I was planning on masking sys-fs/udev but thanks to reading the latest news I know I need to mask sys-apps/systemd-utils instead. The road is getting rocky already but we’ll move forward regardless. Being a fan of the s6 suite of software and mdevd being a device manager by the same author I thought I’d go with that but it’s not available in the main repository. No worries, mdev provided by busybox can work too.

The guide I followed for this part is the excellent gentoo wiki article about mdev. Armed with some echo 'sys-apps/busybox mdev static' > /etc/portage/package.use/busybox we have a device manager. To be honest I forgot about static initially so I ran emerge -avuDN @world so the USE changes would be in effect but without the correct flags for busybox. After I noticed I was missing the static flag, I added it and re-updated. Such is life sometimes. Running emerge -avc removed most of the undesireables but sys-apps/systemd-utils persisted. When trying to see what was keeping it here this is what I got:

emerge -avc systemd-utils

Calculating dependencies... done!
  sys-apps/systemd-utils-250.7 pulled in by:
    virtual/tmpfiles-0-r3 requires sys-apps/systemd-utils[tmpfiles]


emerge -avc virtual/tmpfiles

Calculating dependencies... done!
  virtual/tmpfiles-0-r3 pulled in by:
    sys-apps/man-db-2.10.2-r1 requires virtual/tmpfiles, =virtual/tmpfiles-0-r3
    sys-apps/openrc-0.44.10 requires virtual/tmpfiles, =virtual/tmpfiles-0-r3

So it seems like man and openrc depend on it which is unfortunate but they at least depend on a virtual package so that if a systemd-free options exists we can use that instead. I could get rid of man, it’s not crucial, but that gets me nowhere since openrc also depends on tmpfiles. There is an alternative implementation that used to be the default by gentoo called opentmpfiles that isn’t active anymore but should work and is packaged by funtoo so there is an ebuild for it already. Writing a modified virtual/tmpfiles ebuild to include this is a TODO for later. There is also an implementation in rust. At any rate I seem to have gotten rid of udev which is a pretty big deal. To make sure we’re alerted in case something tries to reinstall it, let’s mask it. I made a file /etc/portage/package.mask/udev with the following contents:

virtual/udev
sys-fs/udev
sys-fs/eudev
sys-fs/udev-init-scripts
virtual/libudev
dev-libs/libgudev
dev-python/pyudev

Continuing with the chroot

Now that that’s handled, where was I? In the new system, right. Setting the timezone and locale and… oh right! I forgot to mask the packages earlier.

Masking undesired software

After setting the timezone and locale I remembered that I only masked udev. We need to mask systemd, dbus, polkit, pam, networkmanager, pulseaudio, lvm, btrfs and wayland.

First up /etc/portage/package.mask/systemd:

app-admin/systemdgenie
dev-ml/systemd
dev-python/python-systemd
sys-apps/gentoo-systemd-integration
sys-apps/systemd
sys-apps/systemd-bootchart
sys-apps/systemd-readahead
sys-apps/systemd-tmpfiles
#sys-apps/systemd-utils #masked. for now...
sys-boot/systemd-boot
sys-kernel/installkernel-systemd-boot
sys-process/systemd-cron

Next up, /etc/portage/package.mask/dbus:

dev-cpp/sdbus-c++
dev-dotnet/ndesk-dbus
dev-dotnet/ndesk-dbus-glib
dev-haskell/dbus
dev-libs/dbus-c++
dev-libs/dbus-glib
dev-libs/libdbusmenu
dev-libs/libdbusmenu-qt
dev-libs/libmodbus
dev-perl/Net-DBus
dev-python/dbus-next
dev-python/dbus-python
dev-python/pydbus
dev-python/python-dbus-next
dev-python/python-dbusmock
dev-qt/qdbus
dev-qt/qdbusviewer
dev-qt/qtdbus
dev-util/dbus-test-runner
dev-util/gdbus-codegen
gnustep-libs/dbuskit
kde-frameworks/kdbusaddons
net-im/skype-dbus-mock
net-libs/dleyna-connector-dbus
sec-policy/selinux-dbus
sys-apps/dbus
sys-apps/dbus-broker
sys-apps/xdg-dbus-proxy

Moving on to /etc/portage/package.mask/polkit:

gnome-extra/polkit-gnome
kde-plasma/polkit-kde-agent
mate-extra/mate-polkit
sys-auth/polkit
sys-auth/polkit-pkla-compat
sys-auth/polkit-qt
lxqt-base/lxqt-policykit
sec-policy/selinux-policykit

Can’t forget about /etc/portage/package.mask/pam:

dev-erlang/epam
dev-ml/opam
dev-ml/opam-client
dev-ml/opam-core
dev-ml/opam-file-format
dev-ml/opam-format
dev-ml/opam-installer
dev-ml/opam-repository
dev-ml/opam-solver
dev-ml/opam-state
dev-perl/Authen-PAM
dev-php/pecl-pam
dev-python/python-pam
kde-plasma/kwallet-pam
net-mail/checkpassword-pam
sci-biology/paml
sys-auth/google-authenticator-libpam-hardened
sys-auth/nss-pam-ldapd
sys-auth/pam-gnupg
sys-auth/pam-pgsql
sys-auth/pam-script
sys-auth/pam_abl
sys-auth/pam_dotfile
sys-auth/pam_fprint
sys-auth/pam_krb5
sys-auth/pam_ldap
sys-auth/pam_mktemp
sys-auth/pam_mount
sys-auth/pam_mysql
sys-auth/pam_p11
sys-auth/pam_require
sys-auth/pam_skey
sys-auth/pam_smb
sys-auth/pam_ssh
sys-auth/pam_ssh_agent_auth
sys-auth/pam_u2f
sys-auth/pam_yubico
sys-auth/pambase
sys-libs/pam
sys-libs/pam_wrapper

Of course /etc/portage/package.mask/networkmanager:

kde-frameworks/networkmanager-qt
net-misc/networkmanager
net-vpn/networkmanager-fortisslvpn
net-vpn/networkmanager-l2tp
net-vpn/networkmanager-libreswan
net-vpn/networkmanager-openconnect
net-vpn/networkmanager-openvpn
net-vpn/networkmanager-pptp
net-vpn/networkmanager-sstp
net-vpn/networkmanager-strongswan
net-vpn/networkmanager-vpnc
sec-policy/selinux-networkmanager
gnome-extra/nm-applet
gui-apps/nm-tray

Getting there /etc/portage/package.mask/pulseaudio:

ev-python/pulsectl
media-libs/libpulse
media-libs/pulseaudio-qt
media-plugins/gst-plugins-pulse
#media-sound/apulse #we might need this
media-sound/pulseaudio
media-sound/pulseaudio-ctl
media-sound/pulseaudio-daemon
media-sound/pulseaudio-modules-bt
media-sound/pulseaudio-virtualmic
media-sound/pulseeffects
media-sound/pulsemixer
net-misc/pulseaudio-dlna
sec-policy/selinux-pulseaudio
xfce-extra/xfce4-pulseaudio-plugin
xfce-extra/xfce4-volumed-pulse

After that, /etc/portage/package.mask/lvm:

1
2

app-backup/mylvmbackup
sys-fs/lvm2

Followed up by /etc/portage/package.mask/btrfs:

app-backup/grub-btrfs
sys-fs/btrfs-heatmap
sys-fs/btrfs-progs
sys-fs/btrfsmaintenance
sys-fs/python-btrfs

Move b get out the way /etc/portage/package.mask/wayland:

app-misc/wayland-utils
dev-cpp/waylandpp
dev-libs/plasma-wayland-protocols
dev-libs/wayland
dev-libs/wayland-protocols
dev-qt/qtwayland
dev-qt/qtwaylandscanner
dev-util/wayland-scanner
gui-apps/wayland-logout
gui-libs/egl-wayland
kde-frameworks/kwayland
kde-plasma/kwayland-integration
kde-plasma/kwayland-server
x11-apps/xisxwayland
x11-base/xwayland

Last and loneliest /etc/portage/package.mask/logind:

`1`	`sys-auth/elogind`

This should about just about cover most of our bases. I might have gone a bit overboard in a few cases but not without reason. I’d rather block something related to the unwanted software and then find out I want to use it and need to comment it out than have to comb through all packages for any offenders.

Colonel trouble

Alright then with that handled we can install a kernel, a bootloader, fix up /etc/fstab and be on our merry way. Sounds easy enough, right? Right? This is coloring outside the lines expert edition, don’t be naive. If you think you should go for one of the distribution kernels, don’t or else dracut comes knocking on your door:

emerge -av gentoo-kernel-bin

These are the packages that would be merged, in order:

Calculating dependencies... done!

!!! All ebuilds that could satisfy "virtual/udev" have been masked.
!!! One of the following masked packages is required to complete your request:
- virtual/udev-217-r5::gentoo (masked by: package.mask)
- virtual/udev-217-r3::gentoo (masked by: package.mask)

(dependency required by "sys-kernel/dracut-056-r1::gentoo" [ebuild])
(dependency required by "sys-kernel/gentoo-kernel-bin-5.15.59::gentoo[initramfs]" [ebuild])
(dependency required by "gentoo-kernel-bin" [argument])
For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook.

This seems easy enough to fix, echo 'sys-kernel/gentoo-kernel-bin -initramfs' > /etc/portage/package.use/gentoo-kernel-bin and we get:

emerge -av gentoo-kernel-bin

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N     ] dev-libs/elfutils-0.187::gentoo  USE="bzip2 nls utils -lzma -static-libs -test (-threads) -valgrind -verify-sig -zstd" 9,027 KiB
[ebuild  N     ] sys-devel/bc-1.07.1-r4::gentoo  USE="readline -libedit -static" 411 KiB
[ebuild  N     ] virtual/libelf-3-r1:0/1::gentoo  0 KiB
[ebuild  N     ] sys-kernel/gentoo-kernel-bin-5.15.59:5.15.59::gentoo  USE="-initramfs -test" 190,647 KiB
[ebuild  N     ] virtual/dist-kernel-5.15.59:0/5.15.59::gentoo  0 KiB

Total: 5 packages (5 new), Size of downloads: 200,083 KiB

Would you like to merge these packages? [Yes/No]

Bootloader

It’s 3am right now and I’d rather not fight a bootloader. I’m giving up on rEFInd for the moment and installing grub.

Booting into the system

Of all the things that could go wrong, XFS is what bit me. I might have forgotten to install something or gentoo-kernel-bin doesn’t support XFS. After that I booted back into the live ISO, mounted the partitions, entered into the system, installed xfsprogs, rebooted and still no dice. The solution that feels better to me is to just reinstall and grab the opportunity to script this as well. I plan on doing that eventually but today is not the day so I went looking for a diffferent solution.

Changing the filesystem

A little scavenging online led me to a tool that says it can convert partitions from one filesystem to another, called fstransform. Since the laptop’s eMMC drive had ubuntu on it, I booted into there, installed git (which is one of the first things I usually install so you can tell how much I use that ubuntu install), build-essential, g++ and automake then compiled and ran the software according to the instructions in the README. With my fingers crossed, I tried booting into gentoo again and… SUCCESS! The fstab is wrong since I forgot to change it from XFS to ext4 so some errors came up and the partition UUID is different which caused a few more errors to appear but I booted into a tty. No matter that I never set a root password meaning I couldn’t log in and didn’t set up networking, this is still great news.

Small configuration and tools

After rebooting to the gentoo installation media hopefully for the last time, I did some common basic setup. This section of the installation did not lead to any issues but I’ll briefly cover it anyway.

`/etc/fstab`

First order of business was

Networking

I set the hostname in /etc/conf.d/hostname, set the usb ethernet interface to dhcp in /etc/conf.d/net, added a couple alternate names for 127.0.0.1 in /etc/hosts and installed dhcpcd.

System tools

The suggested defaults of sysklogd and cronie are fine with me so I installed them along with chrony and ntp for time sync. I also installed htop as well as sudo. Something I forgot was that pam provides su and since we’re not using it, the shadow implementation of su can be used instead. I had to do echo 'sys-apps/shadow su' > /etc/portage/package.use/shadow and then update again.

Package management extras

I prefer eix for searching, equery and other utilites from gentoolkit are very useful and eclean-kernel is just good manners so I got all of them installed. Don’t forget to run eix-update.

User

A non-root user is good security practice and some things won’t even run as root so a little useradd -m -G users,wheel,audio,video,usb,cdrom -s /bin/bash angle && passwd angle sets us up.

Ready, set, go!

With all of that complete, we’re ready to reboot. When doing so, a couple sh errors from mdev show up and neither the wired or wireless interfaces show up. Back into the live media we go. Time to do the dance: install linux-firmware, re-emerge and emerge --config the kernel, remake the grub config and try again. Reboot again and the problem persists. I suspect the problem has to do with the kernel at this point, possibly because I disabled initramfs. Serves me right for trying to take a shortcut instead of just writing a proper kernel config.

Investigation

After a bit of digging, it turns out it’s the module loading. I had to modprobe r8152 which is the right one for my usb ethernet adapter and then it showed up. With this discovery we can say goodbye to the installation media and continue in our journey.

Graphical environment

Now that we can successfully boot, let’s try to get an X environment. According to the gentoo wiki Xorg guide we need to set INPUT_DEVICES and VIDEO_CARDS in make.conf. I know my laptop uses the intel i915 driver for graphics so I’ll set the value to that and for input I’ll leave libinput since it’s there by default.

Installing Xorg

Let’s try emerging xorg-server now:

emerge -pv xorg-server

These are the packages that would be merged, in order:

Calculating dependencies... done!

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by media-libs/mesa-22.1.3::gentoo
# required by media-libs/libepoxy-1.5.10-r1::gentoo[egl]
# required by x11-base/xorg-server-21.1.4::gentoo[-minimal]
# required by x11-drivers/xf86-input-libinput-1.2.1::gentoo
# required by x11-base/xorg-drivers-21.1::gentoo[input_devices_libinput]
>=media-libs/libglvnd-1.4.0 X
# required by x11-base/xorg-drivers-21.1::gentoo[input_devices_libinput,-input_devices_evdev]
>=x11-base/xorg-server-21.1.4 udev
# required by x11-drivers/xf86-video-intel-2.99.917_p20201215::gentoo
# required by x11-base/xorg-drivers-21.1::gentoo[video_cards_i915]
# required by x11-base/xorg-server-21.1.4::gentoo[xorg]
# required by x11-drivers/xf86-input-libinput-1.2.1::gentoo
>=x11-libs/libdrm-2.4.112 video_cards_intel

 * In order to avoid wasting time, backtracking has terminated early
 * due to the above autounmask change(s). The --autounmask-backtrack=y
 * option can be used to force further backtracking, but there is no
 * guarantee that it will produce a solution.

!!! All ebuilds that could satisfy "virtual/libudev:=" have been masked.
!!! One of the following masked packages is required to complete your request:
- virtual/libudev-232-r7::gentoo (masked by: package.mask)
- virtual/libudev-232-r5::gentoo (masked by: package.mask)

(dependency required by "dev-libs/libinput-1.21.0-r1::gentoo" [ebuild])
(dependency required by "x11-drivers/xf86-input-libinput-1.2.1::gentoo" [ebuild])
(dependency required by "x11-base/xorg-drivers-21.1::gentoo[input_devices_libinput]" [ebuild])
(dependency required by "x11-base/xorg-server-21.1.4::gentoo[xorg]" [ebuild])
(dependency required by "x11-drivers/xf86-video-intel-2.99.917_p20201215::gentoo[dri]" [ebuild])
For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook.

Hmm, this doesn’t look very nice. The wiki article suggested tryint to emerge xorg-drivers on its own if the suggested settings don’t work. I’m sure they didn’t mean it this way but it’s a good suggestion for troubleshooting.

emerge -pv xorg-drivers

These are the packages that would be merged, in order:

Calculating dependencies... done!

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by x11-base/xorg-drivers-21.1::gentoo[-input_devices_evdev,input_devices_libinput]
# required by xorg-drivers (argument)
>=x11-base/xorg-server-21.1.4 udev
# required by x11-drivers/xf86-video-intel-2.99.917_p20201215::gentoo
# required by x11-base/xorg-drivers-21.1::gentoo[video_cards_i915]
# required by x11-base/xorg-server-21.1.4::gentoo[xorg]
# required by x11-drivers/xf86-input-libinput-1.2.1::gentoo
>=x11-libs/libdrm-2.4.112 video_cards_intel
# required by media-libs/mesa-22.1.3::gentoo
# required by media-libs/libepoxy-1.5.10-r1::gentoo[egl]
# required by x11-base/xorg-server-21.1.4::gentoo[-minimal]
# required by x11-drivers/xf86-input-libinput-1.2.1::gentoo
# required by x11-base/xorg-drivers-21.1::gentoo[input_devices_libinput]
# required by xorg-drivers (argument)
>=media-libs/libglvnd-1.4.0 X

 * In order to avoid wasting time, backtracking has terminated early
 * due to the above autounmask change(s). The --autounmask-backtrack=y
 * option can be used to force further backtracking, but there is no
 * guarantee that it will produce a solution.

!!! All ebuilds that could satisfy "virtual/libudev:=" have been masked.
!!! One of the following masked packages is required to complete your request:
- virtual/libudev-232-r7::gentoo (masked by: package.mask)
- virtual/libudev-232-r5::gentoo (masked by: package.mask)

(dependency required by "x11-base/xorg-server-21.1.4::gentoo[udev]" [ebuild])
(dependency required by "x11-base/xorg-drivers-21.1::gentoo[-input_devices_evdev,input_devices_libinput]" [ebuild])
(dependency required by "xorg-drivers" [argument])
For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook.

This looks a little simpler to make sense of so let’s analyze the dependency chain:

we’re trying to install xorg-drivers with libinput as an input device
which depends on xf86-input-libinput
which depends on libinput
which depends on libudev and udev
and also requires xorg-server with the udev use flag Since we need our input devices to work and that seems to require libinput, we should look at what exactly the xf86-input-libinput package wants since we’ll need to install that for sure. The ebuild can be found on gentoo’s gitweb instance and says it needs libinput version 1.11.0 or higher. At this point I’ll admit I cheated and tried to solve it before writing down the steps I took but this was the lightbulb moment. Since this isn’t the first time I’ve attempted this I have an ebuild of libinput 1.15.5 that includes a patch from KISS Linux to make the dependency on udev optional. The libinput version is pretty old and the patch has not been updated since but this solution might just work. We can create a local ebuild repository according to this gentoo wiki article. Install eselect-repository and then run eselect repository create local. Then I ran mkdir -p /var/db/repos/local/dev-libs/libinput/files to create all directories I’ll need at once. I also installed pkgdev although it’s not needed for this specific ebuild. Then I downloaded the ebuild, the patch and the manifest using:

1
2
3

curl -o /var/db/repos/local/dev-libs/libinput/libinput-1.15.5.ebuild https://gitlab.com/insanitywholesale/inherently-overlay/-/raw/master/dev-libs/libinput/libinput-1.15.5.ebuild
curl -o /var/db/repos/local/dev-libs/libinput/files/libinput-optional-udev.patch https://gitlab.com/insanitywholesale/inherently-overlay/-/raw/master/dev-libs/libinput/files/libinput-optional-udev.patch
curl -o /var/db/repos/local/dev-libs/libinput/Manifest https://gitlab.com/insanitywholesale/inherently-overlay/-/raw/master/dev-libs/libinput/Manifest`

To test if everything well we can pretend to emerge the exact version of libinput:

emerge -pv '=dev-libs/libinput-1.15.5'

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N     ] sys-libs/mtdev-1.1.6::gentoo  290 KiB
[ebuild  N     ] dev-libs/libevdev-1.12.1::gentoo  USE="-doc -test" 437 KiB
[ebuild  N     ] dev-libs/libinput-1.15.5:0/10::local  USE="-doc -test" INPUT_DEVICES="-wacom" 570 KiB

Total: 3 packages (3 new), Size of downloads: 1,295 KiB

Then we can mask the newer versions using echo '>dev-libs/libinput-1.15.5' > /etc/portage/package.mask/libinput But if we try installing xorg-drivers we see not all is well:

emerge -pv xorg-drivers

These are the packages that would be merged, in order:

Calculating dependencies... done!

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by x11-base/xorg-drivers-21.1::gentoo[input_devices_libinput,-input_devices_evdev]
# required by xorg-drivers (argument)
>=x11-base/xorg-server-21.1.4 udev
# required by x11-drivers/xf86-video-intel-2.99.917_p20201215::gentoo
# required by x11-base/xorg-drivers-21.1::gentoo[video_cards_i915]
# required by x11-base/xorg-server-21.1.4::gentoo[xorg]
# required by x11-drivers/xf86-input-libinput-1.2.1::gentoo
>=x11-libs/libdrm-2.4.112 video_cards_intel
# required by media-libs/mesa-22.1.3::gentoo
# required by media-libs/libepoxy-1.5.10-r1::gentoo[egl]
# required by x11-base/xorg-server-21.1.4::gentoo[-minimal]
# required by x11-drivers/xf86-input-libinput-1.2.1::gentoo
# required by x11-base/xorg-drivers-21.1::gentoo[input_devices_libinput]
# required by xorg-drivers (argument)
>=media-libs/libglvnd-1.4.0 X

 * In order to avoid wasting time, backtracking has terminated early
 * due to the above autounmask change(s). The --autounmask-backtrack=y
 * option can be used to force further backtracking, but there is no
 * guarantee that it will produce a solution.

!!! All ebuilds that could satisfy "virtual/libudev:=" have been masked.
!!! One of the following masked packages is required to complete your request:
- virtual/libudev-232-r7::gentoo (masked by: package.mask)
- virtual/libudev-232-r5::gentoo (masked by: package.mask)

(dependency required by "x11-base/xorg-server-21.1.4::gentoo[udev]" [ebuild])
(dependency required by "x11-base/xorg-drivers-21.1::gentoo[input_devices_libinput,-input_devices_evdev]" [ebuild])
(dependency required by "xorg-drivers" [argument])
For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook.

The problem here is that the xorg-drivers ebuild which you can read here forces xorg-server to have the udev USE flag enabled. We’ll have to go through a similar procedure as before to fix this. Run mkdir -p /var/db/repos/local/x11-base/xorg-drivers to create the directory structure for this package and then

`1`	`curl -o /var/db/repos/local/x11-base/xorg-drivers/xorg-drivers-21.1.ebuild https://gitweb.gentoo.org/repo/gentoo.git/plain/x11-base/xorg-drivers/xorg-drivers-21.1.ebuild`

Following that, delete lines 61 and 66 and then cd /var/db/repos/local/x11-base/xorg-drivers && pkgdev manifest. In my case the local ebuild repository has higher priority so now when trying to install xorg-server get:

Calculating dependencies... done!
[ebuild  N     ] x11-misc/util-macros-1.19.3::gentoo  83 KiB
[ebuild  N     ] x11-misc/xbitmaps-1.1.2-r1::gentoo  127 KiB
[ebuild  N     ] sys-devel/llvm-common-14.0.6::gentoo  USE="-verify-sig" 103,143 KiB
[ebuild  N     ] x11-libs/pixman-0.40.0::gentoo  USE="(-loongson2f) -static-libs -test" CPU_FLAGS_X86="mmxext sse2 -ssse3" 620 KiB
[ebuild  N     ] x11-misc/xkeyboard-config-2.36::gentoo  861 KiB
[ebuild  N     ] gui-libs/display-manager-init-1.0-r4::gentoo  0 KiB
[ebuild  N     ] sys-libs/mtdev-1.1.6::gentoo  290 KiB
[ebuild  N     ] sys-libs/binutils-libs-2.38-r2:0/2.38::gentoo  USE="nls -64-bit-bfd (-cet) -multitarget -static-libs" 23,287 KiB
[ebuild  N     ] app-crypt/rhash-1.4.2::gentoo  USE="nls ssl -debug -static-libs" 408 KiB
[ebuild  N     ] dev-libs/jsoncpp-1.9.5:0/25::gentoo  USE="-doc -test" 211 KiB
[ebuild  N     ] dev-python/pygments-2.12.0-r1::gentoo  USE="-test" PYTHON_TARGETS="python3_10 (-pypy3) -python3_8 -python3_9 (-python3_11)" 4,182 KiB
[ebuild  N     ] dev-python/mako-1.2.1::gentoo  USE="-doc -test" PYTHON_TARGETS="python3_10 (-pypy3) -python3_8 -python3_9 (-python3_11)" 479 KiB
[ebuild  N     ] x11-libs/libXext-1.3.4::gentoo  USE="-doc" 380 KiB
[ebuild  N     ] media-fonts/font-util-1.3.3::gentoo  140 KiB
[ebuild  N     ] x11-libs/libxkbfile-1.1.0::gentoo  357 KiB
[ebuild  N     ] x11-libs/libxshmfence-1.3-r2::gentoo  302 KiB
[ebuild  N     ] x11-libs/libXfixes-6.0.0::gentoo  USE="-doc" 291 KiB
[ebuild  N     ] x11-apps/iceauth-1.0.9::gentoo  128 KiB
[ebuild  N     ] x11-apps/rgb-1.0.6-r1::gentoo  136 KiB
[ebuild  N     ] x11-libs/libxcvt-0.1.2::gentoo  10 KiB
[ebuild  N     ] dev-libs/libevdev-1.12.1::gentoo  USE="-doc -test" 437 KiB
[ebuild  N     ] x11-libs/libXrender-0.9.10-r2::gentoo  302 KiB
[ebuild  N     ] app-arch/libarchive-3.6.1:0/13::gentoo  USE="acl bzip2 e2fsprogs iconv lzma xattr -blake2 -expat -lz4 -lzo -nettle -static-libs -verify-sig -zstd" 7,258 KiB
[ebuild  N     ] dev-libs/libuv-1.44.1:0/1::gentoo  1,272 KiB
[ebuild  N     ] x11-libs/libfontenc-1.1.4::gentoo  313 KiB
[ebuild  N     ] dev-python/docutils-0.19::gentoo  PYTHON_TARGETS="python3_10 (-pypy3) -python3_8 -python3_9 (-python3_11)" 2,009 KiB
[ebuild  N     ] x11-libs/libpciaccess-0.16-r1::gentoo  USE="zlib" 359 KiB
[ebuild  N     ] media-libs/libglvnd-1.4.0::gentoo  USE="X -test" 551 KiB
[ebuild  N     ] x11-libs/libXmu-1.1.3::gentoo  USE="ipv6 -doc" 386 KiB
[ebuild  N     ] x11-apps/xkbcomp-1.4.5::gentoo  246 KiB
[ebuild  N     ] x11-libs/libXfont2-2.0.5::gentoo  USE="bzip2 ipv6 -doc -truetype" 513 KiB
[ebuild  N     ] x11-libs/libXScrnSaver-1.2.3::gentoo  USE="-doc" 285 KiB
[ebuild  N     ] dev-libs/libinput-1.15.5:0/10::local  USE="-doc -test" INPUT_DEVICES="-wacom" 570 KiB
[ebuild  N     ] x11-libs/libXxf86vm-1.1.4-r2::gentoo  USE="-doc" 289 KiB
[ebuild  N     ] x11-libs/libXrandr-1.5.2::gentoo  USE="-doc" 323 KiB
[ebuild  N     ] dev-util/cmake-3.22.4::gentoo  USE="ncurses -doc -emacs -qt5 -test" 9,553 KiB
[ebuild  N     ] x11-libs/libdrm-2.4.112::gentoo  USE="-valgrind" VIDEO_CARDS="intel -amdgpu (-exynos) (-freedreno) -nouveau (-omap) -radeon (-tegra) (-vc4) (-vivante) -vmware" 442 KiB
[ebuild  N     ] x11-apps/xauth-1.1.2::gentoo  154 KiB
[ebuild  N     ] x11-apps/xrdb-1.2.1::gentoo  140 KiB
[ebuild  N     ] x11-apps/xinit-1.4.1-r1::gentoo  USE="-twm" 173 KiB
[ebuild  N     ] sys-devel/llvm-14.0.6-r2:14::gentoo  USE="binutils-plugin libffi ncurses -debug -doc -exegesis -libedit -test -verify-sig -xar -xml -z3" LLVM_TARGETS="(AArch64) (AMDGPU) (ARM) (AVR) (BPF) (Hexagon) (Lanai) (MSP430) (Mips)
(NVPTX) (PowerPC) (RISCV) (Sparc) (SystemZ) (VE) (WebAssembly) (X86) (XCore) (-ARC) (-CSKY) (-M68k)" 229 KiB
[ebuild  N     ] sys-devel/llvmgold-14::gentoo  0 KiB
[ebuild  N     ] media-libs/mesa-22.1.3::gentoo  USE="X gles2 llvm zstd -d3d9 -debug -gles1 -lm-sensors -opencl -osmesa (-selinux) -test -unwind -vaapi -valgrind -vdpau -vulkan -vulkan-overlay -wayland -xa -xvmc -zink" CPU_FLAGS_X86="sse2" VIDEO_CARDS="(-freedreno) -intel (-lima) -nouveau (-panfrost) -r300 -r600 -radeon -radeonsi (-v3d) (-vc4) -virgl (-vivante) -vmware" 15,642 KiB
[ebuild  N     ] media-libs/libepoxy-1.5.10-r1::gentoo  USE="X egl -test" 325 KiB
[ebuild  N     ] x11-base/xorg-server-21.1.4:0/21.1.4::gentoo  USE="xorg -debug -elogind -minimal (-selinux) -suid -systemd -test -udev -unwind -xcsecurity -xephyr -xnest -xvfb" 4,825 KiB
[ebuild  N     ] x11-base/xorg-drivers-21.1::local  INPUT_DEVICES="libinput -elographics -evdev -joystick -synaptics -vmmouse -void -wacom" VIDEO_CARDS="i915 -amdgpu -ast -dummy -fbdev (-freedreno) (-geode) -glint -intel -mga -nouveau -nv
-nvidia (-omap) -qxl -r128 -radeon -radeonsi -siliconmotion (-tegra) (-vc4) -vesa -via -virtualbox -vmware" 0 KiB
[ebuild  N     ] x11-drivers/xf86-input-libinput-1.2.1::gentoo  306 KiB
[ebuild  N     ] x11-drivers/xf86-video-intel-2.99.917_p20201215::gentoo  USE="dri sna -debug -tools -udev -uxa -xvmc" 1,222 KiB

Total: 48 packages (48 new), Size of downloads: 183,533 KiB

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by media-libs/mesa-22.1.3::gentoo
# required by media-libs/libepoxy-1.5.10-r1::gentoo[egl]
# required by x11-base/xorg-server-21.1.4::gentoo[-minimal]
# required by x11-drivers/xf86-input-libinput-1.2.1::gentoo
# required by x11-base/xorg-drivers-21.1::local[input_devices_libinput]
>=media-libs/libglvnd-1.4.0 X
# required by x11-drivers/xf86-video-intel-2.99.917_p20201215::gentoo
# required by x11-base/xorg-drivers-21.1::local[video_cards_i915]
# required by x11-base/xorg-server-21.1.4::gentoo[xorg]
# required by x11-drivers/xf86-input-libinput-1.2.1::gentoo
>=x11-libs/libdrm-2.4.112 video_cards_intel

 * In order to avoid wasting time, backtracking has terminated early
 * due to the above autounmask change(s). The --autounmask-backtrack=y
 * option can be used to force further backtracking, but there is no
 * guarantee that it will produce a solution.

Which is awesome! Just to be safe, I did echo '>x11-base/xorg-drivers-21.1' > /etc/portage/package.mask/xorg-drivers before installing xorg-server. Now we’ll do emerge -av instead of emerge -pv so the USE changes are staged and then etc-update to finalize them before installing xorg-server. Additionally, echo 'x11-base/xorg-server suid' > /etc/portage/package.use/xorg-server because without (e)logind we need that to be able to start an X session.

And now we can finally install xorg-server. Since it pulls in llvm I left it to install overnight.

Starting Xorg

Since git is installed, I pulled down my customized versions of dwm and st to have a window manager and a terminal for testing. This required libXft and libXinerama to be installed since they’re build dependencies. After a quick echo 'exec dwm' > ~/.xinitrc and startx dwm started up successfully! Keyboard and mouse don’t seem to be working though.

Input Devices

The gentoo-kernel-bin package got me this far but now it’s a bottleneck. To remedy the situation I described previously I configured a custom kernel and left it to compile. The important bit was to disable modules so everything is built-in and we don’t need to worry about module loading and all the cruft that comes with that. Keeping in mind that this was done on a dual-core Celeron running at a maximum of 1.6GHz, it takes a while even with the unnecessary things disabled.

After recompiling the kernel and rebooting into it, the same problem persists. This made me turn my attention to what input drivers I’m using. I assumed libinput would do the trick but it doesn’t seem to be that simple. Next thing to try is enable synaptics by changing INPUT_DEVICES="libinput" to INPUT_DEVICES="libinput synaptics" in /etc/portage/make.conf and run an update. After rebooting and running startx nothing seems to have changed.

QEMU Guest Agent for Terraform on ProxMox

Wed, 27 Jul 2022 13:38:18 +0300

In a previous post I talked about how I set up proxmox in order to be able to provision virtual machines on it using terraform. Since then, I’ve improved the setup by adding qemu-guest-agent to the template images without needing to create them myself from scratch and I’d like to share it here.

The Problem

A lot images intended for openstack or cloud use, be they debian or ubuntu or something else, do not include qemu-guest-agent by default. This means that we either don’t get to enable that integration or have to do a complex dance of provisioning the VMs with the agent disabled, installing qemu-guest-agent, turning them off, applying a terraform plan with the agent enabled and booting them back up. Obviously that’s very tedious, error-prone and time-consuming especially when creating and destroying potentially dozens of VMs for testing. To figure out a solution, I went down a bit of a rabbit hole and considered a few solutions before settling on one so I’d like to take you through the selection process.

The Solutions

There are multiple ways to solve this problem and there isn’t really a standard go-to recommended one. One of these might work better for your situation and that’s great however I will be detailing why I didn’t choose them.

Images with qemu-guest-agent pre-installed

This was my first instinct, just find images that have it already. Unfortunately I couldn’t find a debian image that fit this criteria and overall it’s not a reliable solution, some distros might simply elect to not include it.

Terraform remote-exec provisioner

I found this great blog post by Besmir Zanaj which was really interesting and demonstrates how to use terraform to achieve post-provision VM customization. It is worth checking out however for something so basic that all virtual machines need I decided it was better to bake it into the template rather than have the action run on each VM individually. The example from the post about firewall rules is a great use-case for this functionality so it’s not to be discarded entirely.

Packer proxmox-iso Builder

Packer is another tool from Hashicorp, the same company behind terraform, that can create images to be used with proxmox and then make templates based on those images. The functionality is offered by a plugin, specifically a builder, called proxmox-iso. It didn’t appear to be a terrible choice but the configuration examples that I found relied on launching ubuntu, waiting a certain amount of time then sending keystrokes to begin a no-cloud autoinstall. This seemed to me to be very error-prone and distro-specific so I skipped it. I don’t believe it to be a complete dead-end and it’s possible there is a better way to use it for what I want but I couldn’t figure out how.

Adding qemu-guest-agent using guestfs-tools

The last thing I looked into was using guestfs-tools to modify the image before creating a proxmox template out of it. Guest-fs tools is a set of tools for modifying virtual machine disk images that works on a lot of distros. After lots of searching, I came across an awesome blog post by Austin Pivarnik which tackled basically the same problem. As detailed in the post itself, on debian-based systems which luckily proxmox is, we can install libguestfs-tools and then use the virt-customize command to alter the contents of a virtual machine image. How I added this to my setup:

add libguestfs-tools to the list of packages installed by my proxmox setup ansible playbook
add a task (adapted for all VM images) to the same playbook after downloading the image but before creating the template
run playbook
ta-da! You can see the changes I made in this commit where I also adjusted my terraform configuration and a couple other small things. I’ve been very happy with this so far and aside from showing the guest IP in the proxmox web interface as well as allowing smooth shutdown it has also improved provisioning times.

Conclusion

Having good templates for virtual machines on a virtualization platform is very important for automation. While it took quite a long time for me to find a suitable solution I am quite happy with where I ended up and the improvement to my homelab. Thank you for reading and I hope you learned something new.

Network Design Document

Sat, 16 Jul 2022 18:45:44 +0300

I recently started remaking my network at home and decided to publish some of the design documentation here

Network Layout

VDSL Internet comes in through an RJ11 port to a Vodafone H300s modem-router. LAN port 2 of the H300s is plugged into port 2 (untagged VLAN 10) of the Netgear GS724Tv4 switch. Switch port 23 (untagged VLAN 1, tagged in all other VLANs) is plugged into the gigabit ethernet port of the pfsense router. All devices are plugged directly into the switch since they fit. Inter-VLAN routing happens through pfsense and the switch operates as a managed L2 switch even though it has L3 capabilities.

VLANs

Probably overkill but certainly enough:

VLAN	Description	Subnet	Gateway	DHCP range from pfsense
1	Default	-		-
2	Auto VoIP	-	-	-
3	Auto-Video	-	-	-
10	Border Router LAN	10.0.10.0/24	10.0.10.1	-
20	End Devices	10.0.20.0/24	10.0.20.254	10.0.20.10-245
22	Trusted	10.0.20.0/24	10.0.22.254	10.0.22.22-222
30	Monitoring	10.0.30.0/24	10.0.30.254	-
40	Storage	10.0.40.0/24	10.0.40.254	-
50	Servers	10.0.50.0/24	10.0.50.254	-
60	Guest	10.0.60.0/24	10.0.60.254	10.0.60.2-252
65	Media	10.0.65.0/24	10.0.65.254	10.0.65.2-252
66	IoT	10.0.66.0/24	10.0.66.254	10.0.66.2-252
67	DIY IoT	10.0.67.0/24	10.0.67.254	10.0.67.2-252
70	Security	10.0.70.0/24	10.0.70.254	-
80	DMZ	10.0.80.0/24	10.0.80.254	-
90	APIs	10.0.90.0/24	10.0.90.254	-
99	Management	10.99.0.0/16	10.99.0.1	-
111	Kubernetes	10.0.111.0/24	10.0.111.254	-
113	MAAS	10.0.113.0/24	10.0.113.254	-
123	Windows	10.0.123.0/24	10.0.123.254	-
150	Test LAN	-	-	-
666	Native	-	-	-
1022	Unused Port	-	-	-
1337	Danger	-	-	-

Conclusion

That’s my home LAN currently, I’ll try to keep this updated as I change stuff but no promises.

Ansible, Terraform and Proxmox

Fri, 24 Jun 2022 23:24:57 +0300

In the continuing quest to improve my homelab I’ve written a few ansible playbooks as well as terraform configuration to automate a lot of tasks. The initial setup of proxmox is handled by ansible, the provisioning of virtual machines is handled by terraform and the installation of k3s on the VMs is done by ansible.

Proxmox Setup with Ansible

Before getting to use all the shiny devops with a selfhosted solution, we need to do a bit of setup first. This isn’t perfect since it’s just ansible running a bash script but it gets the job done. The other issue,, is that a lot of cloud images don’t include qemu-guest-agent and other packages by default. I have a few ideas how to solve that (check out this TODO.md for more info) but haven’t tested out anything yet. Since qemu-guest-agent and things like nfs-common are required on most VMs, a lot of redundant package installation happens but such is life for now. With that out of the way, let’s start with how I configure proxmox.

Host

While proxmox has a lot of things ready to go out of the box, a bit of configuration is always required and even more so in this case. The goal of the following steps is to turn our hosts into capable, cloud-like, targets for provisioning resources using terraform.

Repository

Initially the enterprise repository which we don’t have a license to access is enabled. To fix this, I delete the file in /etc/apt/sources.list.d that contains it and add a file with the no-subscription repository. This might not be necessary since the lae.proxmox ansible role can do essentially the same thing and many more. Here are the ansible tasks:

    - name: remove enterprise repo
      file:
        path: /etc/apt/sources.list.d/pve-enterprise.list
        state: absent

    - name: add no-subscription repo
      copy:
        src: pve-no-sub.list
        dest: /etc/apt/sources.list.d/pve-no-subscription.list
        mode: '0644'
        owner: root
        group: root

Template VM

The terraform provider for proxmox can work with cloud-init which is a great vendor-agnostic way of configuring the basics (username, password, IP addressing, DNS, SSH keys) in a virtualized environment. This is achieved by having a virtual disk or virtual cd-rom that mounts inside the VM and provides this information to the guest.

VM Image

Since software is not always compatible with the latest version of a linux distribution, in this case debian, I keep the last two versions around. The first step is to download the images:

    - name: get debian 10 cloud image
      get_url:
        url: https://cloud.debian.org/images/cloud/buster/20211011-792/debian-10-generic-amd64-20211011-792.qcow2
        dest: /root/debian-10-openstack-amd64.qcow2
        checksum: sha512:f3dac13104b4e28eb62c46cbbb1e30fc9c792834500f9101d477c19c258c31ff04850933ee0b3e63236eca38c854447d95a0ab45163c4a3fccf05f9dd95601a5
        mode: '0644'
        owner: root
        group: root

    - name: get debian 11 cloud image
      get_url:
        url: https://cloud.debian.org/images/cloud/bullseye/20220121-894/debian-11-generic-amd64-20220121-894.qcow2
        dest: /root/debian-11-openstack-amd64.qcow2
        checksum: sha512:0948dc56b4834a7755e4eae7d5532e138b90484a949161ffe9fc6894c7a14b1bd32ebf96fa3f3d03d498fe7ee125f8014f31bfab5825915a93de9330df814f7b
        mode: '0644'
        owner: root
        group: root

Template Script

As mentioned in the beginning I’d like to automate this with ansible but this is what I have for now:

if [ -z "$1" ]; then
    VMID=9001
else
    VMID="$1"
fi

if [ -z "$2" ]; then
    VMNAME="debian-tmpl"
else
    VMNAME="$2"
fi

VMEXISTS="$(qm list | grep $VMID)"

if [ "$VMEXISTS" ]; then
    echo "VM with ${VMID} already exists"
else
    qm create "${VMID}" -name "${VMNAME}" -memory 1024 -net0 virtio,bridge=vmbr0 -cores 1 -sockets 1 -cpu cputype=kvm64 -description "debbie image" -kvm 1 -numa 1
    qm importdisk "${VMID}" debian-10-openstack-amd64.qcow2 local-zfs
    qm set "${VMID}" -scsihw virtio-scsi-pci -virtio0 local-zfs:vm-"${VMID}"-disk-0
    qm set "${VMID}" -serial0 socket
    qm set "${VMID}" -boot c -bootdisk virtio0
#   qm set "${VMID}" -agent 1 #disabled since VM does not initially have qga installed
    qm set "${VMID}" -hotplug disk,network,usb,memory,cpu
    qm set "${VMID}" -vcpus 1
    qm set "${VMID}" -vga qxl
    qm set "${VMID}" -name "${VMNAME}"
    qm set "${VMID}" -ide2 local-zfs:cloudinit
    qm template "${VMID}"
fi

This works well enough for first-time setup and with a few additions could become idempotent. The above script and its debian 11 equivalent are copied over to the proxmox hosts and executed by the following ansible tasks:

    - name: get debian 10 template vm script
      copy:
        src: setup-debian-10-cloudinit.sh
        dest: /root/setup-debian-10-template.sh
        mode: '0755'
        owner: root
        group: root

    - name: run debian 10 template vm script
      shell:
        cmd: /root/setup-debian-10-template.sh

    - name: get debian 11 template vm script
      copy:
        src: setup-debian-11-cloudinit.sh
        dest: /root/setup-debian-11-template.sh
        mode: '0755'
        owner: root
        group: root

    - name: run debian 11 template vm script
      shell:
        cmd: /root/setup-debian-11-template.sh

Terraform Setup

With proxmox ready, let’s move on to the terraform setup. If you haven’t used it before, terraform is a declarative way of describing our infrastructure. It can manage virtual machines, DNS records, object storage buckets, kubernetes resources and many more things with the appropriate providers.

Provider

Providers in terraform are plugins that enable terraform to manage resources that it doesn’t support out of the box. In this case, we want to manage proxmox virtual machines and containers but it could be other things like a router running pfsense or vyos. In previous versions, the provider would need to be installed manually but nowadays we only need to create a main.tf with the provider listed and upon running terraform init, the provider will be downloaded automatically.

Basic `main.tf`

Let’s create that file then. Open your favorite editor, paste the following and replace the IP as well as the password with the ones applicable to your setup:

terraform {
	required_providers {
		proxmox = {
			source = "Telmate/proxmox"
			version = ">=2.9.5"
		}
	}
}

provider "proxmox" {
	alias = "pve"
	pm_tls_insecure = true
	pm_api_url = "https://192.168.70.1:8006/api2/json" #Replace IP
	pm_password = "password123" #Replace password
	pm_user = "root@pam"
}

Save the file and then run terraform init. You should see the following output (the version might be different):

Initializing the backend...

Initializing provider plugins...
- Finding telmate/proxmox versions matching ">= 2.9.5"...
- Installing telmate/proxmox v2.9.10...
- Installed telmate/proxmox v2.9.10 (self-signed, key ID A9EBBE091B35AFCE)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

And ta-da! We’ve installed the provider and are ready to start creating resources.

Creating a test VM

The above file won’t do anything so we’ll declare a virtual machine and then tell terraform to create it. The example below is specific to my setup so you probably want to modify it before using but here it is:

terraform {
	required_providers {
		proxmox = {
			source = "Telmate/proxmox"
			version = ">=2.9.5"
		}
	}
}

provider "proxmox" {
	alias = "pve"
	pm_tls_insecure = true
	pm_api_url = "https://192.168.70.1:8006/api2/json" #Replace IP
	pm_password = "password123" #Replace password
	pm_user = "root@pam"
}

resource "proxmox_vm_qemu" "proxmox_test_vm" {
	provider = proxmox.pve
	name = "deb-10-test"
	target_node = "pve"

	/* change to debian-templ for 11 */
	clone = "debian-tmpl"
	os_type = "cloud-init"
	cores = 2
	sockets = 1
	cpu = "host"
	memory = 2048
	scsihw = "virtio-scsi-pci"
	bootdisk = "virtio0"
	agent = 0

	disk {
		size = "30G"
		type = "virtio"
		storage = "local-zfs"
	}

	network {
		model = "virtio"
		bridge = "vmbr0"
	}

	ipconfig0 = "ip=192.168.50.25/16,gw=192.168.0.1"

	/*workstation SSH key*/
	sshkeys = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5jzKi37jm3517bqThbw+7LR/GXm3qC6Az5F+ZUa36vYM7Ygk2K5bWcFIL2YUCrkL5jfSsvoowONjCAxyuoyxtW4MJxnQLyq4u4yDsRC7YvBPAKZUYaHwnbkCfDs5a75dEFOoDxCA0DY2GrhqzBndaTcCfl0fZ4vN+9LcKOb1dSKiHeHvsh35YNtwntbL21meo+hiycUEgGwNe9/4kxKpdGTr7HvbeX2Fjm/UZBZIJKVcGop/3gCHXYnKH+OY5zc8cmt9Jg4CIwEqrSKeOX0bE8LSPRpVRXH4v8OcMaMei/HQejlH8NBwybEdJ4mhl8vHaFEjDbIWoOujmiRQF2263 angle@puddle"

	/*required otherwise the state always appears modified*/
	lifecycle {
		ignore_changes = [
			cipassword,
			network,
			desc,
		]
	}
}

After saving the file, we can run terraform plan -out create-test-vm.plan to calculate the changes required and save them to a file. Following that, we can run terraform apply create-test-vm.plan to apply the changes to our infrastructure. Some output will appear while the template is cloned, the VM settings are adjusted and the cloud-init configuration is put into the drive. In the end, you should have a brand new VM created by terraform!

Creating VMs for k3s

The above example is realistic but what if we need 5 similar VMs that could be used as kubernetes nodes? Conveniently, terraform provides a count variable that we can use to slightly differ settings where needed. Here is the resource that provisions the virtual machines that I run a k3s cluster on:

resource "proxmox_vm_qemu" "proxmox_vm_k3s" {
	provider = proxmox.pve
	count = 5
	name = "deb-k3s-${count.index + 1}"
	target_node = "pve"

	clone = "debian-tmpl"
	os_type = "cloud-init"
	cores = 2
	sockets = 1
	cpu = "host"
	memory = 3084
	scsihw = "virtio-scsi-pci"
	bootdisk = "virtio0"
	agent = 0
	onboot = true

	disk {
		size = "30G"
		type = "virtio"
		storage = "local-zfs"
	}

	network {
		model = "virtio"
		bridge = "vmbr0"
	}

	ipconfig0 = "ip=192.168.15.5${count.index + 1}/16,gw=192.168.0.1"

	sshkeys = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5jzKi37jm3517bqThbw+7LR/GXm3qC6Az5F+ZUa36vYM7Ygk2K5bWcFIL2YUCrkL5jfSsvoowONjCAxyuoyxtW4MJxnQLyq4u4yDsRC7YvBPAKZUYaHwnbkCfDs5a75dEFOoDxCA0DY2GrhqzBndaTcCfl0fZ4vN+9LcKOb1dSKiHeHvsh35YNtwntbL21meo+hiycUEgGwNe9/4kxKpdGTr7HvbeX2Fjm/UZBZIJKVcGop/3gCHXYnKH+OY5zc8cmt9Jg4CIwEqrSKeOX0bE8LSPRpVRXH4v8OcMaMei/HQejlH8NBwybEdJ4mhl8vHaFEjDbIWoOujmiRQF2263 angle@puddle"

	lifecycle {
		ignore_changes = [
			cipassword,
			network,
			desc,
		]
	}
}

There aren’t many changes from the previous example, the name and the IP address use the index in order to avoid 5 VMs having the same name and IP address and that’s it!

Conclusion

Hopefully this gives you inspiration for some awesome things you can do to automate your infrastructure and make working in a virtualized environment easier. Thank you for reading.

Cisco SR2024C Repair

Mon, 28 Mar 2022 18:03:46 +0300

One of my friends at university gave me a broken switch that seemed broken and I decided to fix it. Join me on a pretty entertaining troubleshooting path that was not entirely necessary.

Backstory

This semester in university I am taking a CCNA class. The learning path is split across two semesters, next semester I’ll take CCNA 2 to be ready to get the certification. A few of my friends are also there an one of them happened to have a supposedly broken Cisco switch. Great opportunity to use it for hands-on learning, I thought. He said it wasn’t fully broken, it could turn on fine but would turn off after 10-20 minutes. Not that bad. A little fan was also missing from it but that would be no more than 5-10 euros to replace. He gave it to me one day, I took it home and that was that. After a bit of research I realized the switch wasn’t managed meaning it didn’t have Cisco iOS capabilities. Nevertheless, I like tech necromancy and network gear seldom goes unused. So that’s basically how I got a Cisco SR2024C.

Initial Findings

After checking that the power plug looked fine and the ports didn’t look damaged, I decided to turn it on. Indeed, after roughly 20 minutes it started boot-looping so the behavior was as described. The switch wouldn’t completely shut off, instead it followed a little pattern:

lights flash for a second the way they do when first plugged in
lights go out same as when normally booting
goto step 1

This meant that eventually anything plugged into it would drop off the network which is not good. At this point I’m suspecting the power supply since the device is pretty old, about 7 years past its End-Of-Life date.

Open Wide

Behavior confirmed and reproducible, we have a place to start from. The device would need to be opened anyway in order to replace the fan and I’d like to inspect the inside too. This specific switch is considered compact as noted by the ending C in its model number (SR2024C). Its width is a bit more than the width of the ports, 24 RJ45 gigabit ethernet ports and two miniGBIC. It has holes to attach rackmount ears if desired but I wasn’t provided those. On the front there is a plastic cover that fits over the 4 sets of ports (3 sets of 8xRJ45 and one set of the 2 miniGBIC) and also clips on to the top and bottom metal pieces. I dinged the plastic since I used a pair of scissors to pry it off but that’s alright, we’re going for functional not pretty. Following that, I had to carefully unhook some little metal tabs that held the top half of the metal case to the bottom half. Push a little here, pull a little there and ta-da! We’re in. The metal tabs were annoying to hook and unhook so I used a pair of pliers to twist them off. On the left side there is a mesh cutout big enough for a 40mm fan and on the right side it’s almost entirely mesh. Weird for a rackmountable device to have side-to-side airflow but what do I know. The bottom is entirely bare and so is the top with the exception of a Cisco logo.

Guts

There are two PCBs/boards inside, the main board and an open-frame power supply. They are connected by a weird cable I’ve never seen before and have connectors I’ve never seen before. On the PSU side it’s a 6-pin and a smaller 2-pin that merge into an 8-pin on the main board side. The power plug is attached to the PSU through a 2-pin cable with a similar connector as well as a chassis ground point. One good thing is that they all look removable and fairly easy to probe with a multimeter. Overall The internals appear understandable so I’m fairly confident I can at least identify the issue.

Power

Even though I commented about the ease of probing with a multimeter, I’m sad to say that I didn’t have one previous to this experience. Since I mostly work with software, either configuring or writing it, there was never a real need to own a multimeter. I hopped on to one of the local electronic hardware supply stores’ website and found one for around 15 euros, the UNI-T UT131C. It arrived the next morning so I got to work and confirmed what repair videos from similar switches said, the main board uses 5V power. The thing is, in those other videos the voltage was lower and that was what was causing issues. In my case it was nice and steady at 4,98V which is within margin of error meaning nowhere near enough to cause a malfunction. You can never be certain though so I had ordered some alligator clips which I used for powering the main board from an ATX PSU’s molex connector. It didn’t result in a normal boot but I at least got a power light. This ruled out my main hypothesis this far which was that the switch’s power supply was going bad.

Accidental Solution

In the process of hooking the switch up to the ATX power supply, I had it open for troubleshooting. Perhaps accidentally, just to confirm that the problem was still there I reconnected the built-in power supply again. After leaving for an hour or so to do something else, I came back and saw it wasn’t boot looping.

What?

Peculiar, I thought. Sure enough, plugging in my laptop to it showed a link light and the laptop got an IP address. What the… WAIT! By hovering over it I could feel the power supply putting out some heat. No way it could be this simple, right? Could it be overheating? I turned it off, closed the case and put a 120mm fan right up against the right side to pull air out of the switch. To my surprise that was adequate to keep it cool enough even while closed. This was so embarassingly simple that I had to test again. I turned the switch on, let it get to the point where it starts constantly resetting and then put the fan next to it. After about 5 minutes it started operating completely fine, I was in awe of how simply the problem was.

Fan Replacement

Moving on to a permanent solution, I needed a fan to replace then one that was missing. Looking things up online, I realized I need a 5V 1W fan that is 40mm wide and 20mm thick. However, th only thing I found was a 0.75W fan for 8,23 euros. Good enough I guess I’ll just get two of those, one for intake and one for exhaust. Funny thing about plans, the store had only one of the 0.75W version and then one that was 0.54W. Sigh… Fine, I’ll see how I’ll arrange them. A day later they came in, I tested them both out to see that they worked and put in the 0.75W fan just to be sure the internals get enough airflow. After a 24-hour testing period, I think it’s safe to say that the problem is solved.

Use Case

The current plan is to use the switch for a VLAN so I don’t have to waste multiple ports from my main 24-port switch and be reconfiguring them. Yes, port-based VLANs aren’t ideal but I need a place to test my next network remake without bothering everything else. Even with the new fan it’s not very quiet probably due to the whine that small fans make but I have plenty of noisy things in my room.

Conclusion

It was a long and weird journey but it ended well and I now have a new switch. The troubleshooting was fun, as it always is, the new tools are fun to have and I got a lesson about assuming the problem is more complicated than it is. Thank you for reading, I hope you enjoyed the journey cause I sure did.

How This Website Is Made

Sat, 20 Nov 2021 20:56:19 +0200

Intro

A significant part of having my own site is the way it’s made behind the scenes. This has changed in a couple different ways for the duration of its existence and in this post I’ll try to explain how and why.

Beginning

When I first bought inherently.xyz to host my own website I wanted my site to be built in the way that I’d like other sites to be built. The main requirements were:

a theme that was mine
basically no javascript
nothing loading from third-party places
code that is easy to understand
readable in the devtools
reasonably small and fast
able to be read on a lot of devices and browsers
over the top way to host it

I don’t think I have a repository where I developed it but this is an archive of some version of it. I’ll retell the story of its existence so you don’t have to look at the source but it’s there if you want to take a look.

Evaluation

In my humble opinion I did fairly well to cover the requirements I had but not as well as I could have. However not all was great. For one, I had used bootstrap, the CSS framework, to make the site. Because I used the navbar hamburger/button/dropdown thing it required some javascript/jquery which was unfortunate. It was my goal to get rid of this but from reading a github issue about it the team behind bootstrap did not feel like providing that functionality through a CSS trick was something they were willing to do.

The theme was mostly mine outside of boostrap and it was adapted from someone else’s CSS colorscheme. As for the code itself, every page consisted of hand-written HTML so that part was easy to read but the minified bootstrap files, aside from being pretty damn big, were unreadable. At least the bootstrap files were hosted on my own server and no third-party anything loading so I can check that off the list. Additionally, to make the site I had created a file called base.html containing the basic template of a page which I would manually copy to create a new page and then rsync the files from my computer to the server. Surprisingly enough, this worked pretty well although changing code that touched multiple pages like the navigation was somewhat cumbersome. Reading the website through a lesser known web browser like netsurf, surf or any text-mode web browser was possible so that requirement was basically 100% satisfied.

Since the beginning of this site I’ve been using docker-compose to host it. For this iteration I was using a simple nginx image and mapping the directory on the filesystem to /usr/share/nginx/html inside the container. Traefik was running in front of it acting as a reverse proxy and to support https. Overkill for a site that has handwritten HTML? Check.

Improvements

There was quite a bit of way to go as you can gather from the above. The main pain points were the existence of javascript causing the navigation dropdown to not work on mobile when javascript was disabled, the performance metrics related to using bootstrap in general, dependency on bootstrap and lack of major customization from me. The raw HTML workflow was mildly annoying but it never stopped me from writing anything.

Solutions

I spent quite a long time trying to find a workaround for the nav dropdown and eventually stumbled into a solution in a post I can’t find anymore, I’ll update this if I do. That worked fine enough and allowed me to get rid of the jquery dependency that, funnily enough, had a vulnerability announced about it while I was rewriting my nav code. Bootstrap was also slowing down everything according to all benchmarks I ran.

This was all fairly fine for quite a while. I kept working on the site, writing stuff and having my own little space on the internet. The curse of being into tech and knowing how to make a website even as basic as this was that I wanted to rewrite it. That’s when I started exploring and found out about hugo, a fast static site generator (SSG for short) written in Go. Given that I was interested in Go at the time of this rewrite and that I knew how the templates worked more or less I decided to give it a try.

Hugo

My first idea was to just port it to hugo and then improve it and quickly realized what a mistake that would be. Why do one thing you’ve never done before if you can do two, right? Hugo is a static site generator meaning that it creates the files required for the site once and that’s it, the site is not dynamically put together on the client. As the author, you mostly write markdown files that are turned into html at build time and then you can put those html files anywhere you’d like. This is what the site uses to this day and you find the source code for it here.

Transitioning

The real work started when I decided I’d make my own theme too. I didn’t just have to move my content and page structure to it but write some nice fresh CSS too since I decided I’ve had enough of bootstrap.

Directory Structure

The first steps was examining the directory structure and finding the most low level template file which is layouts/_default/baseof.html and making my way up. Inside there I defined the very basics of what the HTML would look like, imported my CSS file which at this point had about 5 lines in it and moved on to the other fundamental pages. These are layouts/_default/single.html and layouts/_default/list.html, the former being about a “one of it’s kind” page and the latter being an aggregation. The example here is a blog post and the list of all the blog posts. They are based on baseof.html and in my case are very short, consisting only of the barebones required to implement these types of pages. This is because they are overwritten by category-specific single.html and list.html which is how I make blog posts have a table of contents and how I make the blog section look different than the tutorials section.

Layouts

Speaking of sections, the first thing I got up and running was the blog since that was the easiest one with lots of documentation and examples. The second priority was the front page which turned out to be a little bit more complicated. You put the content for it inside content/_index.md and the layout-related stuff inside layouts/index.html. In hindsight it’s fairly intuitive but it took a few minutes of digging when I was less familiar. For each sub-category you can basically include no layout files in which case the list.html and single.html from layouts/_default will be used or you can write ones that will override them. As an example, in layouts/tutorials I have both a list.html and a single.html to build the tutorial-related pages in their own way.

Layouts are pretty cool and probably my favorite thing is partial layouts. These allow you to put a snippet in a file, let’s say footer.html and then include it using {{ partial "footer" . }} in another template. One other example is the contact form, I just made a layouts/contact/list.html that has the line {{ partial "comments.html" . }} and this way I can keep the template cleaner and allow reusing that comment submission form elsewhere too. So I just went around creating my categories, writing specific list or single templates for them if required and throwing some pieces of reusable code in partial layout templates.

CSS

The most involved part was by far the CSS. I have to say that if it’s not obvious by now I’m not a “frontend person” per se, I just want something that looks clean, fast and readable. As a disclaimer, I’m not a fan of the size of the fonts which are by far the biggest element right now but I’m a huge fan of the fira font family (with the exception of fira code cause I hate ligatures) but they work well enough. It took the majority of the time and had the most updates, tweaks and fixes by far compared to anything else. Essentially nothing from the original CSS is the same except the colors which was a long process but a necessary one to get where I wanted to be. In the end I’m satisfied with it, I think it looks fairly unique and serves its purpose well while coming in at under 200 lines (it is way less in reality but the number balloons up because of the way I format the code).

Docker

This time I went with a custom docker image so I could have more control and no filesystem bind mounts. The main problem is that the hugo team does not publish an official docker image so I had to use the following to get hugo installed in the container in order to generate the site (most current Dockerfile is in the git repository, check there first before copying):

FROM golang:1.16 as buildsite
ENV CGO_ENABLED 0
WORKDIR /go/src
RUN git clone https://github.com/gohugoio/hugo.git
WORKDIR /go/src/hugo
RUN go install -v
WORKDIR /go/src/inheresite-hugo
COPY . .
RUN hugo

FROM nginx:alpine
COPY default.conf /etc/nginx/conf.d/
COPY --from=buildsite /go/src/inheresite-hugo/public /usr/share/nginx/html

This means I compile hugo every time I want to upload a post but it goes fairly fast and gives a bit of time to reflect on what I’ve written. If this is not for you, use either an ubuntu image and install it with apt if you don’t need the latest features or an arch one if you need the latest release for some reason. Using third-party images is a last resort but those are technically also usable for this purpose however I’d advise against using images from random people.

Impressions

After a long while with hugo I think I’ll be sticking with it. It allows for some pretty clean code, I’m partial to Go templating and it has served me well. I’d like to look into some small improvements such as merging tutorials into blog and just displaying them in a different way but that’s a topic for another day. The full server-side syntax highlighting is awesome too, I thought I’d need to cave and use some javascript thing to do it but thankfully it worked out great and I found a code theme that matches the rest of the site. If you want to get started with it, just make sure you know some basic HTML and have some patience, the rest can be figured out in an evening.

Conclusion

The site you’re currently looking at has been a while in the making and there are still a couple items on my TODO.md for it but I’m happy enough with it for now. You might want to check out the comment microservice I wrote for use with the comment form mentioned a bit above, it’s far from perfect but it gets the job done. Thank you for reading this, I hope you enjoyed it and maybe even learned something.

GK61 Review

Sat, 04 Sep 2021 14:49:11 +0300

Intro

A little less than a month ago I bought a new keyboard, the HK Gaming GK61. It’s a 60% mechanical keyboard with RGB backlighting and a few different switch options. I’ve been using full-size membrane keyboards up until now so this was quite a bit different.

Overview

The GK61 comes in black, white, red or pink colors. It can be outfitted with gateron optical black, blue, brown, red, silver or yellow switches. As denoted by the name, its 60% size is a noteworthy selling point. Of course you can’t buy anything without RGB nowadays so it has that too. There is also a piece of software for configuring keyboard settings and saves them on the keyboard itself. I paid 70 euros for it in greece and I bought the black model with blue switches. With the big picture view out of the way, let’s get a bit more into the specifics.

Switches

As previously mentioned, I went with the gateron optical blue switches. This was mostly done because I like the noise and tactile feedback. I have a habit of bottoming out the keys so blues being a little slower for people that only barely activate the switch is not an issue. One potential drawback I was made aware of is that optical switches are not as common so replacing them is potentially more difficult. All the keys work fine, the sound is enjoyable, the physical feedback is good so no real complaints here.

60% layout

The size of the keyboard was a big change for me but also one of the big reasons I chose it. The unfortunate part is that it was only available in ANSI and not ISO locally but I’m aware that an ISO version exists. So what’s the deal with the 60% size? What does that mean exactly? There is no numpad, no F1 to F12, no end, home, pgup, pgdown, insert, delete or arrow keys. What does exist however, is an Function (Fn) key that acts as a modifier. This means that you can press Fn and something else to access the alternative function of a key. It saves size but can also be a bit annoying when you can’t access some keys directly. My personal changes were that I remapped caps lock to escape, right alt to left arrow, right ctrl to right arrow and escape to backtick. I’m slowly finding out that I’d prefer it if there were arrow keys that are just arrow keys on the main layer. Overall I like the small size but I think a 65% size with an ISO layout would be a better option for me.

Software

This is where things get a bit more gloomy. There exists software for the GK61 but not all software utilities are the same. In specific, I got a GK61v2 which has a serial number that starts with GK61US and things get interesting. The earlier versions of this keyboard use seemingly different firmware and different software utilities. Additionally, the alternative key functions are different for the GK61v2 so the M1, M2, M3 mode keys don’t work. At any rate, the software works fine. It allows remapping keys as well as customizing the lighting mode. Well…about that lightning customization… One would think that customizing one of the custom lighting modes would allow it to be permanently saved but no. If you make a custom one and then hit Fn and \ to cycle all of them, your custom mode will be nowhere to be found. This is unfortunate but I don’t mind since I just wanted to have a solid color and move on however it’s a downside worth mentioning. In totality the software side of the keyboard is a downside but it isn’t completely unusable I guess.

Conclusion

After almost a month of use, I’m overall pretty happy with the GK61. For 70 euros it was a fairly big investment, it was my first mechanical keyboard as well as my first 60% keyboard. Ideally I would like to have an ISO layout and separate arrow keys so when I upgrade in the future I’ll look at getting a 65% one with the appropriate key layout or possibly build a custom one. Overall I’d recommend it just be aware that it’s a lot smaller than what you might be used to. Thank you for reading, I hope you got something out of this.

Flux Update

Wed, 25 Aug 2021 00:44:34 +0300

I wrote about my journey of choosing a continuous deployment tool and why I ultimately ended up using flux for my homelab in this post. It took quite a bit of reasearch and if you’re in a similar position as I was it might prove to be helpful.

Long(ish)-term experience

I’ve been using it for about 5 months now, the first post was written about one month of use. At that point I had been tinkering with it and attempting to get all the stuff I wanted to be deployed by it. Some things were simple, others not so straight-forward.

Successes

Deploying raw manifest applications

Deploying raw kubernetes manifests was really easy. Since I’m somewhat on the fence about Helm, I prefer using normal kubernetes manifests where possible. This meant that for my use case mostly everything was smooth sailing after figuring out how to generate a kustomization yaml using the command-line tool.

Command-line tool

Speaking of that, flux has a command-line tool to interact with the controllers running inside the cluster. It’s also the way you can initialize a repository to be used as the place for flux files to be stored. It can also be used to install flux to the cluster if the repository already exists.

Upgrading

When a new version of flux comes out, you can upgrade and it will carry over to the cluster. It’s really easy, it just upgrades its own files so when those are committed it will pull in the new version of the controllers.

Trouble

Aggregation yaml

For starters here is a minor one that took way too long to diagnose. I forget exactly what this is called but essentially it is a yaml file says run deal with the listed resources. Imagine the following directory contents:

drwxrwxr-x 4 angle angle 4,0K Αυγ  25 01:23 .
drwxrwxr-x 5 angle angle 4,0K Ιουλ 21 03:42 ..
drwxrwxr-x 2 angle angle 4,0K Αυγ  13 15:55 helmrepos
-rw-rw-r-- 1 angle angle  219 Ιουλ 21 03:20 infra-source.yml
-rw-rw-r-- 1 angle angle  126 Αυγ  12 11:45 kustomization.yaml
drwxrwxr-x 2 angle angle 4,0K Αυγ  13 15:56 storage

The kustomization.yaml file is as follows:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - infra-source.yml
  - helmrepos
  - storage

So it’s a way to more easily include stuff. The issue I had is that I named the file kustomization.yml instead of kustomization.yaml which for some reason isn’t supported. I believe this is a bug and not intended behavior but it was still frustrating to find and fix.

Logs

Now on to more substantial issues. While troubleshooting I got a bit frustrated with how flux displays errors. The messages are fairly non-descript and there isn’t a lot of guidance for how to fix them.

Conclusion

Despite a couple rough edges, flux has been working incredibly well. Even before I got it fully functioning, it was amazing to be able to just install it to a newly creted cluster and have all my applications running just like that. It’s almost magical and I’ve been enjoying it quite a lot. If you aren’t using a continuous deployment tool, feel free to give flux a try or read the previous post to learn about some other options

Docker Multi-Binary Images

Sun, 01 Aug 2021 17:43:30 +0300

Problems in deploying distributed applications

If you read my background in the front page, you know I’m not a prolific programmer by any definition. This isn’t going to be a gigantic revalation and if you’ve worked with deploying your own applications to kubernetes you’ll already know most of the following things. With that said, what are the problems with deploying a distributed application? I was tinkering with gifinator and while writing the kubernetes manifests I realized there was a problem. A few bugs required multi-service re-deployments. Let’s say there was something affecting the render service and the gifcreator worker service. I could rebuild 2 new docker images, push them to the local image repository, update the version in their respective manifests and apply the changes. The issue is that after doing it once or twice it became really annoying when wanting to roll back to check the previous behavior. The more this cycle went on, I had to keep track of a set of 4 different versions which quickly became unmanageable.

Lightbulb moment

That’s when the obvious dawned on me, the upstream kubernetes manifests all used the same image somehow. I quickly opened them up and checked to see they were redefining the container’s CMD directive! In fact the actual Dockerfile did not even have a CMD nor ENTRYPOINT defined. The code was built, the binaries were copied to the second stage and the final image was shipped. Using this method the final image is as small as possible without needing 4 separate ones.

How do

It’s pretty easy to do this so let’s take a look at the Dockerfile as well as the kubernetes manifests.

Dockerfile

This specific Dockerfile is available on gitlab for those interested but I’ll paste it here for ease:

# build stage
FROM golang:1.16 as build

ENV CGO_ENABLED 0
ENV GOOS linux
ENV GOARCH amd64
ENV GO111MODULE on

WORKDIR /go/src/gifinator
COPY . .

WORKDIR /go/src/gifinator/render
RUN go get -v
RUN go vet -v
RUN go install -v

WORKDIR /go/src/gifinator/gifcreator
RUN go get -v
RUN go vet -v
RUN go install -v

WORKDIR /go/src/gifinator/frontend
RUN go get -v
RUN go vet -v
RUN make installwithvars

# run stage
FROM busybox as run
RUN mkdir /tmp/objcache
RUN mkdir /tmp/scene
COPY --from=build /go/bin/render /render
COPY --from=build /go/bin/gifcreator /gifcreator
COPY --from=build /go/bin/frontend /frontend
COPY ./gifcreator/scene /tmp/scene
COPY ./frontend/templates /templates
ENV FRONTEND_TEMPLATES_DIR=/templates

First up we set some basic environment variables and copy the code into the building environment. Then one by one the services are compiled and installed into the build stage image. Following that, in the run stage the needed directories are created, the binaries are copied over, assets are also copied over and a final environment variable is set. There is no CMD or ENTRYPOINT as you can see, this will probably result in the inability to just docker run -p externalport:internalport imagename but the tradeoff is well worth it.

Kubernetes deployment manifests

With the Dockerfle out of the way, let’s look at the way these images are deployed. The primary way I run containers is through kubernetes so we’ll be looking at kubernetes manifests. Same as the Dockerfile you can find the source for these manifests on gitlab, I’ll be linking each one. kube/manifests/gifinator/render/deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: render
  namespace: gifinator
  labels:
    app: render
spec:
  replicas: 1
  selector:
    matchLabels:
      app: render
  template:
    metadata:
      labels:
        app: render
    spec:
      containers:
      - name: render
        image: inherently/gifinator:0.0.5
        command: ["/render"]
#the rest of the file is omitted

kube/manifests/gifinator/gifcreator-server/deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gifcreator-server
  namespace: gifinator
  labels:
    app: gifcreator-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gifcreator-server
  template:
    metadata:
      labels:
        app: gifcreator-server
    spec:
      containers:
      - name: gifcreator-server
        image: inherently/gifinator:0.0.5
        command: ["/gifcreator"]
#the rest of the file is omitted

kube/manifests/gifinator/gifcreator-worker/deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gifcreator-worker
  namespace: gifinator
  labels:
    app: gifcreator-worker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gifcreator-worker
  template:
    metadata:
      labels:
        app: gifcreator-worker
    spec:
      containers:
      - name: gifcreator-worker
        image: inherently/gifinator:0.0.5
        command: ["/gifcreator", "-worker"]
#the rest of the file is omitted

kube/manifests/gifinator/frontend/deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: gifinator
  labels:
    app: frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        image: inherently/gifinator:0.0.5
        command: ["/frontend"]
#the rest of the file is omitted

Here are 4 different services, all easily using the same image. Just by redifining the command it’s easy to simplify the development process by a quite a lot. It’s worth noting that I’ve only shown the deployment manifests since the other ones don’t have anything to do with the container image used.

Conclusion

This is a short post mostly made up of code snippets but I hope you learned something and can improve your workflow with distributed microservice applications.

Go API Part 3

Sat, 10 Jul 2021 15:56:21 +0300

Previous relevant tutorial

The development of the CRUD REST API was described in part 2 and the introduction to the series can be found in part 1 so read those first if you’re new.

Intro

After creating a minimally functional API, we’re going to add persistent storage to it in the form of a database. In this case it is going to be postgres running inside docker for ease of installation and use.

Starting point

Here is where we left off last time:

// main.go
package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
	"strconv"
)

type DeliveryDriver struct {
	FirstName string `json:"firstname"`
	LastName  string `json:"lastname"`
}

type Delivery struct {
	OrderNumber      int            `json:"ordernumber"`
	City             string         `json:"city"`
	Zipcode          string         `json:"zipcode"`
	Address          string         `json:"address"`
	Phone1           string         `json:"phone1"`
	Phone2           string         `json:"phone2,omitempty"`
	Cancelled        bool           `json:"cancelled"`
	Delivered        bool           `json:"delivered"`
	DeliveryAttempts int            `json:"deliveryattempts"`
	Driver           DeliveryDriver `json:"deliverydriver"`
}

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{
	&Delivery{
		OrderNumber:      1,
		City:             "Here",
		Zipcode:          "52011",
		Address:          "Home",
		Phone1:           "6945123789",
		Phone2:           "2313722903",
		Cancelled: false,
		Delivered:        false,
		DeliveryAttempts: 0,
		Driver: DeliveryDriver{
			FirstName: "Mhtsos",
			LastName:  "Iwannou",
		},
	},
	&Delivery{
		OrderNumber:      2,
		City:             "There",
		Zipcode:          "1701",
		Address:          "Office",
		Phone1:           "6932728091",
		Cancelled: false,
		Delivered:        true,
		DeliveryAttempts: 1,
		Driver: DeliveryDriver{
			FirstName: "Lucas",
			LastName:  "Johnson",
		},
	},
	&Delivery{
		OrderNumber:      3,
		City:             "FarAway",
		Zipcode:          "920639",
		Address:          "Island",
		Phone1:           "6900777123",
		Cancelled: true,
		Delivered:        false,
		DeliveryAttempts: 24,
		Driver: DeliveryDriver{
			FirstName: "Pilotos",
			LastName:  "Aeroplanou",
		},
	},
}

// Constant for Bad Request
const BadReq string = `{"error": "bad request"}`
// Constant for Not Found
const NotFound string = `{"error": "not found"}`

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(deliveries)
}

// Read a specific delivery
func GetDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

// Create a new delivery
func AddDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	d := &Delivery{}
	// because deliveries start from 1
	orderNum := int(len(deliveries)) + 1
	err := json.NewDecoder(r.Body).Decode(d)
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	// Override whatever the sender had sent
	d.OrderNumber = orderNum
	// Append the delivery to deliveries list
	deliveries = append(deliveries, d)
	w.WriteHeader(http.StatusCreated)
	// Send back the delivery that was saved
	json.NewEncoder(w).Encode(d)
	return
}

// Update an existing delivery
func UpdateDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			err := json.NewDecoder(r.Body).Decode(d)
			if err != nil {
				w.WriteHeader(http.StatusBadRequest)
				w.Write([]byte(BadReq))
				return
			}
			d.OrderNumber = orderNum
			w.WriteHeader(http.StatusAccepted)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

// Delete a delivery (not really)
func DeleteDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			d.Cancelled = true
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

func main() {
	// Set up router
	router := mux.NewRouter()
	// Set up subrouter for api version 1
	apiV1 := router.PathPrefix("/api/v1").Subrouter()
	// Set up routes
	apiV1.HandleFunc("/deliveries", GetAllDeliveries).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery/{ordernumber}", GetDelivery).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery", AddDelivery).Methods(http.MethodPost)
	apiV1.HandleFunc("/delivery/{ordernumber}", UpdateDelivery).Methods(http.MethodPut)
	apiV1.HandleFunc("/delivery/{ordernumber}", DeleteDelivery).Methods(http.MethodDelete)
	// Start http server
	log.Fatal(http.ListenAndServe(":8000", router))
}

Restructuring

In my opinion a good first move is to abstract away the specifics of the database before moving further. Some might see it as a premature optimization but I believe it’s important to not mix the concerns of storing data with the API. We will do that by having an interface that many different data storage backends can satisfy. Initially we will remove the specifics of the simple list we used and make it more generalized. Up until now all the code was in a single file and it’s kind of crowded with data models, API routes, API functions and data storage so that will change.

Important note

Since we will be splitting the contents of the main package over different files we won’t be able to run our program using go run main.go. All the .go files need to be included so the command should instead be go run *.go

Models

First off let’s cut and then paste the models in their own file called models.go next to main.go like so:

// models.go
package main

type DeliveryDriver struct {
	FirstName string `json:"firstname"`
	LastName  string `json:"lastname"`
}

type Delivery struct {
	OrderNumber      int            `json:"ordernumber"`
	City             string         `json:"city"`
	Zipcode          string         `json:"zipcode"`
	Address          string         `json:"address"`
	Phone1           string         `json:"phone1"`
	Phone2           string         `json:"phone2,omitempty"`
	Cancelled        bool           `json:"cancelled"`
	Delivered        bool           `json:"delivered"`
	DeliveryAttempts int            `json:"deliveryattempts"`
	Driver           DeliveryDriver `json:"deliverydriver"`
}

Simple enough I’d say. Remember to delete the code from main.go otherwise the compiler will error out. Next let’s handle the API stuff.

API

In a new file called api.go we will first copy the relevant functions and then do a little refactoring like renaming main() since we can’t have two of them as well as making that function return something that can be easily used by the actual main() function. In this case we’ll just return the router and let the code in main.go handle what port it will run at. Once again, remember to delete this code from main.go otherwise it won’t run.

// api.go
package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"net/http"
	"strconv"
)

// Constant for Bad Request
const BadReq string = `{"error": "bad request"}`
// Constant for Not Found
const NotFound string = `{"error": "not found"}`

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(deliveries)
}

// Read a specific delivery
func GetDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

// Create a new delivery
func AddDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	d := &Delivery{}
	// because deliveries start from 1
	orderNum := int(len(deliveries)) + 1
	err := json.NewDecoder(r.Body).Decode(d)
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	// Override whatever the sender had sent
	d.OrderNumber = orderNum
	// Append the delivery to deliveries list
	deliveries = append(deliveries, d)
	w.WriteHeader(http.StatusCreated)
	// Send back the delivery that was saved
	json.NewEncoder(w).Encode(d)
	return
}

// Update an existing delivery
func UpdateDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			err := json.NewDecoder(r.Body).Decode(d)
			if err != nil {
				w.WriteHeader(http.StatusBadRequest)
				w.Write([]byte(BadReq))
				return
			}
			d.OrderNumber = orderNum
			w.WriteHeader(http.StatusAccepted)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

// Delete a delivery (not really)
func DeleteDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			d.Cancelled = true
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

func MakeRouter() http.Handler {
	// Set up router
	router := mux.NewRouter()
	// Set up subrouter for api version 1
	apiV1 := router.PathPrefix("/api/v1").Subrouter()
	// Set up routes
	apiV1.HandleFunc("/deliveries", GetAllDeliveries).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery/{ordernumber}", GetDelivery).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery", AddDelivery).Methods(http.MethodPost)
	apiV1.HandleFunc("/delivery/{ordernumber}", UpdateDelivery).Methods(http.MethodPut)
	apiV1.HandleFunc("/delivery/{ordernumber}", DeleteDelivery).Methods(http.MethodDelete)
	return router
}

Looking nice and neat. We should probably tinker with main.go too though.

Main after API restructure

Let’s adjust main.go to fit with the changes we’ve made. The list stuff is still here (for now) but the line count is greatly reduced:

// main.go
package main

import (
	"log"
	"net/http"
)

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{
	&Delivery{
		OrderNumber:      1,
		City:             "Here",
		Zipcode:          "52011",
		Address:          "Home",
		Phone1:           "6945123789",
		Phone2:           "2313722903",
		Cancelled: false,
		Delivered:        false,
		DeliveryAttempts: 0,
		Driver: DeliveryDriver{
			FirstName: "Mhtsos",
			LastName:  "Iwannou",
		},
	},
	&Delivery{
		OrderNumber:      2,
		City:             "There",
		Zipcode:          "1701",
		Address:          "Office",
		Phone1:           "6932728091",
		Cancelled: false,
		Delivered:        true,
		DeliveryAttempts: 1,
		Driver: DeliveryDriver{
			FirstName: "Lucas",
			LastName:  "Johnson",
		},
	},
	&Delivery{
		OrderNumber:      3,
		City:             "FarAway",
		Zipcode:          "920639",
		Address:          "Island",
		Phone1:           "6900777123",
		Cancelled: true,
		Delivered:        false,
		DeliveryAttempts: 24,
		Driver: DeliveryDriver{
			FirstName: "Pilotos",
			LastName:  "Aeroplanou",
		},
	},
}

func main() {
	router := MakeRouter()
	log.Fatal(http.ListenAndServe(":8000", router))
}

Since we’re cleaning things up and are planning to add a database, time to remove the list and put it in its own file.

List Restructure

Our main.go is getting emptier so let’s remove the list stuff and put it in its own file called listdb.go

// listdb.go
package main

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{
	&Delivery{
		OrderNumber:      1,
		City:             "Here",
		Zipcode:          "52011",
		Address:          "Home",
		Phone1:           "6945123789",
		Phone2:           "2313722903",
		Cancelled: false,
		Delivered:        false,
		DeliveryAttempts: 0,
		Driver: DeliveryDriver{
			FirstName: "Mhtsos",
			LastName:  "Iwannou",
		},
	},
	&Delivery{
		OrderNumber:      2,
		City:             "There",
		Zipcode:          "1701",
		Address:          "Office",
		Phone1:           "6932728091",
		Cancelled: false,
		Delivered:        true,
		DeliveryAttempts: 1,
		Driver: DeliveryDriver{
			FirstName: "Lucas",
			LastName:  "Johnson",
		},
	},
	&Delivery{
		OrderNumber:      3,
		City:             "FarAway",
		Zipcode:          "920639",
		Address:          "Island",
		Phone1:           "6900777123",
		Cancelled: true,
		Delivered:        false,
		DeliveryAttempts: 24,
		Driver: DeliveryDriver{
			FirstName: "Pilotos",
			LastName:  "Aeroplanou",
		},
	},
}

Nice and easy, we radically reduced the code inside main.go and separated our code in nicely named files.

Main after list database restructure

Take a peek at main.go after all the changes. You will notice that it is a lot shorter and only deals with running our API server and not much else:

package main

import (
	"log"
	"net/http"
)

func main() {
	router := MakeRouter()
	log.Fatal(http.ListenAndServe(":8000", router))
}

Database

This is a big addition so we’ll have to think about it a little bit. In my opinion a good first move is to abstract away the specifics of the database. Some might see it as a premature optimization but I believe it’s important to not mix the concerns of storing data with the API. We will do that by having an interface that many different data storage backends can satisfy.

Interface

The fake database we used in part 2 is mostly in its own file called listdb.go but there are still implementation-specific details of it inside api.go. Before ripping out what exists, we’ll define our interface. Due to what it is, it fits best inside models.go. This is a simple interface of something that can essentially perform the same actions as the API except it only deals with talking to the database. Take a look at what is in models.go now:

// models.go
package main

type DeliveryDB interface {
	ReturnAll() ([]*Delivery, error)
	ReturnOne(orderNumber int) (*Delivery, error)
	Store(*Delivery) (*Delivery, error)
	Change(orderNumber int, del *Delivery) (*Delivery, error)
	Remove(orderNumber int) error
}

type DeliveryDriver struct {
	FirstName string `json:"firstname"`
	LastName  string `json:"lastname"`
}

type Delivery struct {
	OrderNumber      int            `json:"ordernumber"`
	City             string         `json:"city"`
	Zipcode          string         `json:"zipcode"`
	Address          string         `json:"address"`
	Phone1           string         `json:"phone1"`
	Phone2           string         `json:"phone2,omitempty"`
	Cancelled        bool           `json:"cancelled"`
	Delivered        bool           `json:"delivered"`
	DeliveryAttempts int            `json:"deliveryattempts"`
	Driver           DeliveryDriver `json:"deliverydriver"`
}

Short and sweet, does what it should. How do we use this abstraction though? For this we’ll need to move quite a few things out of api.go into listdb.go.

Separating concerns

We will start by moving the things related to data storage out of api.go into listdb.go. This will be done so we can implement the DeliveryDB interface. Before the actual work, take a look at a shortened version of the data-related parts of api.go:

// api.go

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	json.NewEncoder(w).Encode(deliveries)
}

// Read a specific delivery
func GetDelivery(w http.ResponseWriter, r *http.Request) {
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
}

// Create a new delivery
func AddDelivery(w http.ResponseWriter, r *http.Request) {
	// Override whatever the sender had sent
	d.OrderNumber = orderNum
	// Append the delivery to deliveries list
	deliveries = append(deliveries, d)
}

// Update an existing delivery
func UpdateDelivery(w http.ResponseWriter, r *http.Request) {
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			err := json.NewDecoder(r.Body).Decode(d)
			if err != nil {
				w.WriteHeader(http.StatusBadRequest)
				w.Write([]byte(BadReq))
				return
			}
			d.OrderNumber = orderNum
			w.WriteHeader(http.StatusAccepted)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
}

// Delete a delivery (not really)
func DeleteDelivery(w http.ResponseWriter, r *http.Request) {
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			d.Cancelled = true
			json.NewEncoder(w).Encode(d)
			return
		}
	}
}

These is what we will grab and reshape to be the implementation of the DeliveryDB interface inside listdb.go. The ReturnAll, ReturnOne, Store, Change and Remove functions should be implemented on a type, in this case, deliveryList.

List database

There isn’t much more to say so here it is:

// listdb.go
package main

import (
	"errors"
)

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{
	&Delivery{
		OrderNumber:      1,
		City:             "Here",
		Zipcode:          "52011",
		Address:          "Home",
		Phone1:           "6945123789",
		Phone2:           "2313722903",
		Cancelled:        false,
		Delivered:        false,
		DeliveryAttempts: 0,
		Driver: DeliveryDriver{
			FirstName: "Mhtsos",
			LastName:  "Iwannou",
		},
	},
	&Delivery{
		OrderNumber:      2,
		City:             "There",
		Zipcode:          "1701",
		Address:          "Office",
		Phone1:           "6932728091",
		Cancelled:        false,
		Delivered:        true,
		DeliveryAttempts: 1,
		Driver: DeliveryDriver{
			FirstName: "Lucas",
			LastName:  "Johnson",
		},
	},
	&Delivery{
		OrderNumber:      3,
		City:             "FarAway",
		Zipcode:          "920639",
		Address:          "Island",
		Phone1:           "6900777123",
		Cancelled:        true,
		Delivered:        false,
		DeliveryAttempts: 24,
		Driver: DeliveryDriver{
			FirstName: "Pilotos",
			LastName:  "Aeroplanou",
		},
	},
}

func NewListDatabase() deliveryList {
	return deliveries
}

func (dl deliveryList) ReturnAll() ([]*Delivery, error) {
	return deliveries, nil
}

func (dl deliveryList) ReturnOne(orderNumber int) (*Delivery, error) {
	for _, d := range deliveries {
		if d.OrderNumber == orderNumber {
			return d, nil
		}
	}
	return nil, errors.New(NotFound)
}

func (dl deliveryList) Store(d *Delivery) (*Delivery, error) {
	d.OrderNumber = int(len(deliveries))
	deliveries = append(deliveries, d)
	return d, nil
}

func (dl deliveryList) Change(orderNumber int, del *Delivery) (*Delivery, error) {
	for _, d := range deliveries {
		if d.OrderNumber == orderNumber {
			del.OrderNumber = orderNumber
			d = del
			return d, nil
		}
	}
	return nil, errors.New(NotFound)
}

func (dl deliveryList) Remove(orderNumber int) error {
	for _, d := range deliveries {
		if d.OrderNumber == orderNumber {
			d.Cancelled = true
			return nil
		}
	}
	return errors.New(NotFound)
}

We’ve moved everything related to data-handling out of the API code and have a mock database to work with. Obviously it’s only in memory so if we stop and restart the application any changes made to the data isn’t saved anywhere.

Refactor API code

Before moving on to implementing a different database let’s change the API code to use the generalized interface. I also changed a couple stuff related to HTTP error codes because some were incorrect and some because most clients expect a 200 response if a request is successful. At any rate, let’s look at the refactored api.go:

package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"net/http"
	"strconv"
)

// Constant for Bad Request
const BadReq string = `{"error": "bad request"}`

// Constant for Not Found
const NotFound string = `{"error": "not found"}`

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	dels, err := deliverydb.ReturnAll()
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(dels)
	return
}

// Read a specific delivery
func GetDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	del, err := deliverydb.ReturnOne(orderNum)
	if err != nil {
		w.WriteHeader(http.StatusNotFound)
		w.Write([]byte(NotFound))
		return
	}
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(del)
	return
}

// Create a new delivery
func AddDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	d := &Delivery{}
	err := json.NewDecoder(r.Body).Decode(d)
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	del, err := deliverydb.Store(d)
	if err != nil {
		w.WriteHeader(http.StatusNotFound)
		w.Write([]byte(NotFound))
		return
	}
	w.WriteHeader(http.StatusCreated)
	json.NewEncoder(w).Encode(del)
	return
}

// Update an existing delivery
func UpdateDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	d := &Delivery{}
	err = json.NewDecoder(r.Body).Decode(d)
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	del, err := deliverydb.Change(orderNum, d)
	if err != nil {
		w.WriteHeader(http.StatusNotFound)
		w.Write([]byte(NotFound))
		return
	}
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(del)
	return
}

// Delete a delivery (not really)
func DeleteDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	err = deliverydb.Remove(orderNum)
	if err != nil {
		w.WriteHeader(http.StatusNotFound)
		w.Write([]byte(NotFound))
		return
	}
	w.WriteHeader(http.StatusOK)
	return
}

func MakeRouter() http.Handler {
	// Set up router
	router := mux.NewRouter()
	// Set up subrouter for api version 1
	apiV1 := router.PathPrefix("/api/v1").Subrouter()
	// Set up routes
	apiV1.HandleFunc("/deliveries", GetAllDeliveries).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery/{ordernumber}", GetDelivery).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery", AddDelivery).Methods(http.MethodPost)
	apiV1.HandleFunc("/delivery/{ordernumber}", UpdateDelivery).Methods(http.MethodPut)
	apiV1.HandleFunc("/delivery/{ordernumber}", DeleteDelivery).Methods(http.MethodDelete)
	return router
}

And here is main.go where the DeliveryDB is initialized:

package main

import (
	"log"
	"net/http"
)

var deliverydb DeliveryDB

func main() {
	// Create a database
	deliverydb = NewListDatabase()
	// Start http server
	router := MakeRouter()
	log.Fatal(http.ListenAndServe(":8000", router))
}

Postgres

There are lots of databases out there but for the purposes of this tutorial I chose postgres. It’s essentially the only one out of the traditional RDBMS systems worth using. The Go standard library is pretty rich so there is a database/sql package although we have to provide a database driver to it. The two most popular drivers for postgres are pq and pgx. Since pq is in maintenance mode and pgx is the recommended alternative by the pq authors we’ll go with that. The guide on the pgx wiki is a great starting point but we’ll add a big more to it.

Connection code

Enough rambling, let’s see how we can start using postgres. Create a file called postgres.go with the following contents:

package main

import (
	"database/sql"
	_ "github.com/jackc/pgx/v4/stdlib"
)

type postgresDB struct {
	client *sql.DB
	pgURL  string
}

func newPostgresClient(url string) (*sql.DB, error) {
	client, err := sql.Open("pgx", url)
	if err != nil {
		return nil, err
	}
	err = client.Ping()
	if err != nil {
		return nil, err
	}
	return client, nil
}

func NewPostgresDB(url string) (*postgresDB, error) {
	pgclient, err := newPostgresClient(url)
	if err != nil {
		return nil, err
	}
	db := &postgresDB{
		pgURL:  url,
		client: pgclient,
	}
	return db, nil
}

func (pdb *postgresDB) ReturnAll() ([]*Delivery, error) {
	return []*Delivery{}, nil
}

func (pdb *postgresDB) ReturnOne(orderNumber int) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Store(d *Delivery) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Change(orderNumber int, del *Delivery) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Remove(orderNumber int) error {
	return nil
}

Quite a few stuff going on. First up, the imports. As discussed we can use database/sql from the standard library and then combine it with a driver. The line _ "github.com/jackc/pgx/v4/stdlib" means that we’re importing that dependency but we won’t have to write stdlib.Something to use its functions and variables but instead just write Something. Following that we have the postgresDB struct which will implement the DeliveryDB interface and which stores the database URL and the database client that uses that URL. Next up, there is the initialization code which is split into two functions, one private and one public as indicated by the lowercase/uppercase letter. The private function, named newPostgresClient creates a client for the database through sql.Open using the pgx driver and then pings the database to make sure it’s accessible. The public function runs newPostgresClient with the provided URL and if there are no errors it saves the URL as well as the client to a new instance of postgresDB and returns it to the caller. Last but not least we see that ReturnAll, ReturnOne, Store, Change and Remove methods are implemented on the postgresDB struct as indicated by (pdb *postgresDB). Right now they don’t really do anything, they’re there just so the DeliveryDB interface is satisfied and the compiler doesn’t exit with an error.

Choose database

Now that we have written code to connect to postgres, we should probably set it up in main.go so if the environment variable PG_URL is set, we use that connection string to connect to postgres.

package main

import (
	"log"
	"net/http"
	"os"
)

var deliverydb DeliveryDB

func main() {
	// Create a database
	pgURL := os.Getenv("PG_URL")
	if pgURL != "" {
		db, err := NewPostgresDB(pgURL)
		if err != nil {
			log.Fatalf("error with postgres connection %v", err)
		}
		deliverydb = db
		log.Println("connected to postgres")
	} else {
		deliverydb = NewListDatabase()
		log.Println("connected to listdb")
	}
	// Start http server
	router := MakeRouter()
	log.Fatal(http.ListenAndServe(":8000", router))
}

It’s pretty simple, we get the value of the PG_URL variable and if it’s not empty, we use it to connect to postgres. If it’s empty we just use the fake list database instead so our web service can still run.

Test connection

While we are confident that we did everything right, it doesn’t hurt to test it out. Using docker we can quickly create a test database without permanently storing any data. The following command should bring up a database we can use for testing:

`1`	`docker run -d --rm --name testpostgres -p 5432:5432 -e POSTGRES_PASSWORD=Apasswd -e POSTGRES_USER=tester postgres:latest`

After it’s done, set the PG_URL environment variable like so:

`1`	`export PG_URL="postgresql://tester:Apasswd@localhost:5432?sslmode=disable"`

And then run the service the same way we’ve been doing all along:

`1`	`go run *.go`

The message connected to postgres should appear in the command line. And there we go, our connection is working and we can move on. Kill the running database with:

`1`	`docker stop testpostgres`

And let’s see how to write some SQL queries.

Queries

In order to work with the database we will need to write queries to be used. For the sake of simplicity we won’t deduplicate the delivery drivers but that is something you could do in case you’re looking for ways to tweak the project on your own. With that in mind, what queries are we going to need? One per method of the delivery database interface should be enough, along with one to create the table on first run.

Create table query

Let’s begin then by creating a file called queries.go with the create table query:

package main

var createDeliveriesTableQuery = `CREATE TABLE IF NOT EXISTS Deliveries (
	OrderNumber SERIAL PRIMARY KEY,
	City VARCHAR,
	Zipcode VARCHAR,
	Address VARCHAR,
	Phone1 VARCHAR,
	Phone2 VARCHAR,
	Cancelled BOOL,
	Delivered BOOL,
	DeliveryAttempts INTEGER,
	DriverFirstName VARCHAR,
	DriverLastName VARCHAR
);`

This is an SQL representation of our Delivery model. One thing you might not have seen is SERIAL PRIMARY KEY. It will automatically increment when a new entry is inserted so we don’t have to keep track of it ourselves. Let’s incorporate this query into the postgres.go code:

package main

import (
	"database/sql"
	_ "github.com/jackc/pgx/v4/stdlib"
)

type postgresDB struct {
	client *sql.DB
	pgURL  string
}

func newPostgresClient(url string) (*sql.DB, error) {
	client, err := sql.Open("pgx", url)
	if err != nil {
		return nil, err
	}
	err = client.Ping()
	if err != nil {
		return nil, err
	}
	_, err = client.Exec(createDeliveriesTableQuery)
	if err != nil {
		return nil, err
	}
	return client, nil
}

func NewPostgresDB(url string) (*postgresDB, error) {
	pgclient, err := newPostgresClient(url)
	if err != nil {
		return nil, err
	}
	db := &postgresDB{
		pgURL:  url,
		client: pgclient,
	}
	return db, nil
}

func (pdb *postgresDB) ReturnAll() ([]*Delivery, error) {
	return []*Delivery{}, nil
}

func (pdb *postgresDB) ReturnOne(orderNumber int) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Store(d *Delivery) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Change(orderNumber int, del *Delivery) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Remove(orderNumber int) error {
	return nil
}

Just 4 new lines and our database will be set up on first connection. You can run the connection test again as we did in the previous section.

See all deliveries

This requires a simple query, we just select everything in the Deliveries table so take a loot at queries.go:

package main

var createDeliveriesTableQuery = `CREATE TABLE IF NOT EXISTS Deliveries (
	OrderNumber SERIAL PRIMARY KEY,
	City VARCHAR,
	Zipcode VARCHAR,
	Address VARCHAR,
	Phone1 VARCHAR,
	Phone2 VARCHAR,
	Cancelled BOOL,
	Delivered BOOL,
	DeliveryAttempts INTEGER,
	DriverFirstName VARCHAR,
	DriverLastName VARCHAR
);`
var retrieveAllDeliveriesQuery = `SELECT 
	OrderNumber,
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName FROM Deliveries;`

And then we’ll use it in the ReturnAll method inside postgres.go

package main

import (
	"database/sql"
	_ "github.com/jackc/pgx/v4/stdlib"
)

type postgresDB struct {
	client *sql.DB
	pgURL  string
}

func newPostgresClient(url string) (*sql.DB, error) {
	client, err := sql.Open("pgx", url)
	if err != nil {
		return nil, err
	}
	err = client.Ping()
	if err != nil {
		return nil, err
	}
	_, err = client.Exec(createDeliveriesTableQuery)
	if err != nil {
		return nil, err
	}
	return client, nil
}

func NewPostgresDB(url string) (*postgresDB, error) {
	pgclient, err := newPostgresClient(url)
	if err != nil {
		return nil, err
	}
	db := &postgresDB{
		pgURL:  url,
		client: pgclient,
	}
	return db, nil
}

func (pdb *postgresDB) ReturnAll() ([]*Delivery, error) {
	var deliveryslice = []*Delivery{}
	rows, err := pdb.client.Query(retrieveAllDeliveriesQuery)
	if err != nil {
		return nil, err
	}
	defer rows.Close()
	for rows.Next() {
		var delivery = &Delivery{}
		err = rows.Scan(
			&delivery.OrderNumber,
			&delivery.City,
			&delivery.Zipcode,
			&delivery.Address,
			&delivery.Phone1,
			&delivery.Phone2,
			&delivery.Cancelled,
			&delivery.Delivered,
			&delivery.DeliveryAttempts,
			&delivery.Driver.FirstName,
			&delivery.Driver.LastName,
		)
		if err != nil {
			return nil, err
		}
		deliveryslice = append(deliveryslice, delivery)
	}
	return deliveryslice, nil
}

func (pdb *postgresDB) ReturnOne(orderNumber int) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Store(d *Delivery) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Change(orderNumber int, del *Delivery) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Remove(orderNumber int) error {
	return nil
}

Initially, we create an empty list of deliveries to store all the deliveries retrieved from the database. Then we run the query, iterate over the rows returned, save the results to the delivery struct and append that struct to the list. Finally we return the list to the caller and that’s all.

See one delivery

This one is also pretty simple, we select everything if the entry has the specified order number:

package main

var createDeliveriesTableQuery = `CREATE TABLE IF NOT EXISTS Deliveries (
	OrderNumber SERIAL PRIMARY KEY,
	City VARCHAR,
	Zipcode VARCHAR,
	Address VARCHAR,
	Phone1 VARCHAR,
	Phone2 VARCHAR,
	Cancelled BOOL,
	Delivered BOOL,
	DeliveryAttempts INTEGER,
	DriverFirstName VARCHAR,
	DriverLastName VARCHAR
);`
var retrieveAllDeliveriesQuery = `SELECT 
	OrderNumber,
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName FROM Deliveries;`
var retrieveOneDeliveryQuery = `SELECT
	OrderNumber,
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName
FROM Deliveries WHERE OrderNumber=$1;`

And now we need to use it inside our ReturnOne method:

package main

import (
	"database/sql"
	_ "github.com/jackc/pgx/v4/stdlib"
)

type postgresDB struct {
	client *sql.DB
	pgURL  string
}

func newPostgresClient(url string) (*sql.DB, error) {
	client, err := sql.Open("pgx", url)
	if err != nil {
		return nil, err
	}
	err = client.Ping()
	if err != nil {
		return nil, err
	}
	_, err = client.Exec(createDeliveriesTableQuery)
	if err != nil {
		return nil, err
	}
	return client, nil
}

func NewPostgresDB(url string) (*postgresDB, error) {
	pgclient, err := newPostgresClient(url)
	if err != nil {
		return nil, err
	}
	db := &postgresDB{
		pgURL:  url,
		client: pgclient,
	}
	return db, nil
}

func (pdb *postgresDB) ReturnAll() ([]*Delivery, error) {
	var deliveryslice = []*Delivery{}
	rows, err := pdb.client.Query(retrieveAllDeliveriesQuery)
	if err != nil {
		return nil, err
	}
	defer rows.Close()
	for rows.Next() {
		var delivery = &Delivery{}
		err = rows.Scan(
			&delivery.OrderNumber,
			&delivery.City,
			&delivery.Zipcode,
			&delivery.Address,
			&delivery.Phone1,
			&delivery.Phone2,
			&delivery.Cancelled,
			&delivery.Delivered,
			&delivery.DeliveryAttempts,
			&delivery.Driver.FirstName,
			&delivery.Driver.LastName,
		)
		if err != nil {
			return nil, err
		}
		deliveryslice = append(deliveryslice, delivery)
	}
	return deliveryslice, nil
}

func (pdb *postgresDB) ReturnOne(orderNumber int) (*Delivery, error) {
	var delivery = &Delivery{}
	err := pdb.client.QueryRow(retrieveOneDeliveryQuery, orderNumber).Scan(
		&delivery.OrderNumber,
		&delivery.City,
		&delivery.Zipcode,
		&delivery.Address,
		&delivery.Phone1,
		&delivery.Phone2,
		&delivery.Cancelled,
		&delivery.Delivered,
		&delivery.DeliveryAttempts,
		&delivery.Driver.FirstName,
		&delivery.Driver.LastName,
	)
	if err != nil {
		return nil, err
	}
	return delivery, nil
}

func (pdb *postgresDB) Store(d *Delivery) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Change(orderNumber int, del *Delivery) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Remove(orderNumber int) error {
	return nil
}

This is basically the same as ReturnAll but even simpler so I won’t cover it in detail.

Add one delivery

In order to test the retrieve methods, we need to have a way to store stuff first. Since the database generates the order number, the query needs to return it when we run it.

package main

var createDeliveriesTableQuery = `CREATE TABLE IF NOT EXISTS Deliveries (
	OrderNumber SERIAL PRIMARY KEY,
	City VARCHAR,
	Zipcode VARCHAR,
	Address VARCHAR,
	Phone1 VARCHAR,
	Phone2 VARCHAR,
	Cancelled BOOL,
	Delivered BOOL,
	DeliveryAttempts INTEGER,
	DriverFirstName VARCHAR,
	DriverLastName VARCHAR
);`
var retrieveAllDeliveriesQuery = `SELECT 
	OrderNumber,
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName FROM Deliveries;`
var retrieveOneDeliveryQuery = `SELECT
	OrderNumber,
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName
FROM Deliveries WHERE OrderNumber=$1;`
var storeDeliveryQuery = `INSERT INTO Deliveries (
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) RETURNING OrderNumber;`

This way when we insert an entry, we will get back its order number. Let’s see how the Store method will be implemented

package main

import (
	"database/sql"
	_ "github.com/jackc/pgx/v4/stdlib"
)

type postgresDB struct {
	client *sql.DB
	pgURL  string
}

func newPostgresClient(url string) (*sql.DB, error) {
	client, err := sql.Open("pgx", url)
	if err != nil {
		return nil, err
	}
	err = client.Ping()
	if err != nil {
		return nil, err
	}
	_, err = client.Exec(createDeliveriesTableQuery)
	if err != nil {
		return nil, err
	}
	return client, nil
}

func NewPostgresDB(url string) (*postgresDB, error) {
	pgclient, err := newPostgresClient(url)
	if err != nil {
		return nil, err
	}
	db := &postgresDB{
		pgURL:  url,
		client: pgclient,
	}
	return db, nil
}

func (pdb *postgresDB) ReturnAll() ([]*Delivery, error) {
	var deliveryslice = []*Delivery{}
	rows, err := pdb.client.Query(retrieveAllDeliveriesQuery)
	if err != nil {
		return nil, err
	}
	defer rows.Close()
	for rows.Next() {
		var delivery = &Delivery{}
		err = rows.Scan(
			&delivery.OrderNumber,
			&delivery.City,
			&delivery.Zipcode,
			&delivery.Address,
			&delivery.Phone1,
			&delivery.Phone2,
			&delivery.Cancelled,
			&delivery.Delivered,
			&delivery.DeliveryAttempts,
			&delivery.Driver.FirstName,
			&delivery.Driver.LastName,
		)
		if err != nil {
			return nil, err
		}
		deliveryslice = append(deliveryslice, delivery)
	}
	return deliveryslice, nil
}

func (pdb *postgresDB) ReturnOne(orderNumber int) (*Delivery, error) {
	var delivery = &Delivery{}
	err := pdb.client.QueryRow(retrieveOneDeliveryQuery, orderNumber).Scan(
		&delivery.OrderNumber,
		&delivery.City,
		&delivery.Zipcode,
		&delivery.Address,
		&delivery.Phone1,
		&delivery.Phone2,
		&delivery.Cancelled,
		&delivery.Delivered,
		&delivery.DeliveryAttempts,
		&delivery.Driver.FirstName,
		&delivery.Driver.LastName,
	)
	if err != nil {
		return nil, err
	}
	return delivery, nil
}

func (pdb *postgresDB) Store(d *Delivery) (*Delivery, error) {
	err := pdb.client.QueryRow(storeDeliveryQuery,
		d.City,
		d.Zipcode,
		d.Address,
		d.Phone1,
		d.Phone2,
		d.Cancelled,
		d.Delivered,
		d.DeliveryAttempts,
		d.Driver.FirstName,
		d.Driver.LastName,
	).Scan(&d.OrderNumber)
	if err != nil {
		return nil, err
	}
	return d, nil
}

func (pdb *postgresDB) Change(orderNumber int, del *Delivery) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Remove(orderNumber int) error {
	return nil
}

In this method we just insert what the user sent, ignoring the order number they specified, and then return the entry as it was saved.

Update one delivery

There are a few ways to go about updating existing entries. We can check each thing about the Delivery that is sent to us and update only the things that are changed. Another alternative is to swap out all existing values with what is sent which is the dumb method. The first option allows for sending less data and maybe saving some bandwidth but the second one is easier to implement so we’ll hope whoever uses our API does so responsibly. Below is queries.go with our update query added. Note that I used some different SQL syntax this time to keep it a little interesting:

package main

var createDeliveriesTableQuery = `CREATE TABLE IF NOT EXISTS Deliveries (
	OrderNumber SERIAL PRIMARY KEY,
	City VARCHAR,
	Zipcode VARCHAR,
	Address VARCHAR,
	Phone1 VARCHAR,
	Phone2 VARCHAR,
	Cancelled BOOL,
	Delivered BOOL,
	DeliveryAttempts INTEGER,
	DriverFirstName VARCHAR,
	DriverLastName VARCHAR
);`
var retrieveAllDeliveriesQuery = `SELECT 
	OrderNumber,
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName FROM Deliveries;`
var retrieveOneDeliveryQuery = `SELECT
	OrderNumber,
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName
FROM Deliveries WHERE OrderNumber=$1;`
var storeDeliveryQuery = `INSERT INTO Deliveries (
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) RETURNING OrderNumber;`
var changeDeliveryQuery = `UPDATE Deliveries SET
	City=$2,
	Zipcode=$3,
	Address=$4,
	Phone1=$5,
	Phone2=$6,
	Cancelled=$7,
	Delivered=$8,
	DeliveryAttempts=$9,
	DriverFirstName=$10
	DriverLastName=$11
WHERE OrderNumber=$1;`

Not too bad I’d say, we just set the values given if the entry has the order number given. Using this in postgres is not too hard since we’ve seen the previous ones, give it a look:

package main

import (
	"database/sql"
	_ "github.com/jackc/pgx/v4/stdlib"
)

type postgresDB struct {
	client *sql.DB
	pgURL  string
}

func newPostgresClient(url string) (*sql.DB, error) {
	client, err := sql.Open("pgx", url)
	if err != nil {
		return nil, err
	}
	err = client.Ping()
	if err != nil {
		return nil, err
	}
	_, err = client.Exec(createDeliveriesTableQuery)
	if err != nil {
		return nil, err
	}
	return client, nil
}

func NewPostgresDB(url string) (*postgresDB, error) {
	pgclient, err := newPostgresClient(url)
	if err != nil {
		return nil, err
	}
	db := &postgresDB{
		pgURL:  url,
		client: pgclient,
	}
	return db, nil
}

func (pdb *postgresDB) ReturnAll() ([]*Delivery, error) {
	var deliveryslice = []*Delivery{}
	rows, err := pdb.client.Query(retrieveAllDeliveriesQuery)
	if err != nil {
		return nil, err
	}
	defer rows.Close()
	for rows.Next() {
		var delivery = &Delivery{}
		err = rows.Scan(
			&delivery.OrderNumber,
			&delivery.City,
			&delivery.Zipcode,
			&delivery.Address,
			&delivery.Phone1,
			&delivery.Phone2,
			&delivery.Cancelled,
			&delivery.Delivered,
			&delivery.DeliveryAttempts,
			&delivery.Driver.FirstName,
			&delivery.Driver.LastName,
		)
		if err != nil {
			return nil, err
		}
		deliveryslice = append(deliveryslice, delivery)
	}
	return deliveryslice, nil
}

func (pdb *postgresDB) ReturnOne(orderNumber int) (*Delivery, error) {
	var delivery = &Delivery{}
	err := pdb.client.QueryRow(retrieveOneDeliveryQuery, orderNumber).Scan(
		&delivery.OrderNumber,
		&delivery.City,
		&delivery.Zipcode,
		&delivery.Address,
		&delivery.Phone1,
		&delivery.Phone2,
		&delivery.Cancelled,
		&delivery.Delivered,
		&delivery.DeliveryAttempts,
		&delivery.Driver.FirstName,
		&delivery.Driver.LastName,
	)
	if err != nil {
		return nil, err
	}
	return delivery, nil
}

func (pdb *postgresDB) Store(d *Delivery) (*Delivery, error) {
	err := pdb.client.QueryRow(storeDeliveryQuery,
		d.City,
		d.Zipcode,
		d.Address,
		d.Phone1,
		d.Phone2,
		d.Cancelled,
		d.Delivered,
		d.DeliveryAttempts,
		d.Driver.FirstName,
		d.Driver.LastName,
	).Scan(&d.OrderNumber)
	if err != nil {
		return nil, err
	}
	return d, nil
}

func (pdb *postgresDB) Change(orderNumber int, del *Delivery) (*Delivery, error) {
	_, err := pdb.client.Exec(changeDeliveryQuery,
		orderNumber,
		del.City,
		del.Zipcode,
		del.Address,
		del.Phone1,
		del.Phone2,
		del.Cancelled,
		del.Delivered,
		del.DeliveryAttempts,
		del.Driver.FirstName,
		del.Driver.LastName,
	)
	if err != nil {
		return nil, err
	}
	return pdb.ReturnOne(orderNumber)
}

func (pdb *postgresDB) Remove(orderNumber int) error {
	return nil
}

As you can see we don’t use the order number from the delivery object since the user might have maliciously changed it. In addition, ReturnOne is called to make sure that we send back the changed entry as it is in the database so there is no data discrepancies between what we store and what we show.

Delete one delivery

If you remember, we don’t actually delete deliveries, we just set the Cancelled status to true. However for the sake of showing how the query would be, I’ll include another one along with its usage inside prostgres as comments alongside the proper one.

package main

var createDeliveriesTableQuery = `CREATE TABLE IF NOT EXISTS Deliveries (
	OrderNumber SERIAL PRIMARY KEY,
	City VARCHAR,
	Zipcode VARCHAR,
	Address VARCHAR,
	Phone1 VARCHAR,
	Phone2 VARCHAR,
	Cancelled BOOL,
	Delivered BOOL,
	DeliveryAttempts INTEGER,
	DriverFirstName VARCHAR,
	DriverLastName VARCHAR
);`
var retrieveAllDeliveriesQuery = `SELECT 
	OrderNumber,
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName FROM Deliveries;`
var retrieveOneDeliveryQuery = `SELECT
	OrderNumber,
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName
FROM Deliveries WHERE OrderNumber=$1;`
var storeDeliveryQuery = `INSERT INTO Deliveries (
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) RETURNING OrderNumber;`
var changeDeliveryQuery = `UPDATE Deliveries SET
	City=$2,
	Zipcode=$3,
	Address=$4,
	Phone1=$5,
	Phone2=$6,
	Cancelled=$7,
	Delivered=$8,
	DeliveryAttempts=$9,
	DriverFirstName=$10
	DriverLastName=$11
WHERE OrderNumber=$1;`
var removeDeliveryQuery = `UPDATE Deliveries SET Cancelled=true WHERE OrderNumber=$1`
//commented out so the compiler doesn't complain about unused variables
//var removeDeliveryForRealQuery = `DELETE FROM Deliveries WHERE OrderNumber=$1`

The query we’ll use is another update, this time it simply sets the Cancelled field to true. Here is the postgres part of it:

package main

import (
	"database/sql"
	_ "github.com/jackc/pgx/v4/stdlib"
)

type postgresDB struct {
	client *sql.DB
	pgURL  string
}

func newPostgresClient(url string) (*sql.DB, error) {
	client, err := sql.Open("pgx", url)
	if err != nil {
		return nil, err
	}
	err = client.Ping()
	if err != nil {
		return nil, err
	}
	_, err = client.Exec(createDeliveriesTableQuery)
	if err != nil {
		return nil, err
	}
	return client, nil
}

func NewPostgresDB(url string) (*postgresDB, error) {
	pgclient, err := newPostgresClient(url)
	if err != nil {
		return nil, err
	}
	db := &postgresDB{
		pgURL:  url,
		client: pgclient,
	}
	return db, nil
}

func (pdb *postgresDB) ReturnAll() ([]*Delivery, error) {
	var deliveryslice = []*Delivery{}
	rows, err := pdb.client.Query(retrieveAllDeliveriesQuery)
	if err != nil {
		return nil, err
	}
	defer rows.Close()
	for rows.Next() {
		var delivery = &Delivery{}
		err = rows.Scan(
			&delivery.OrderNumber,
			&delivery.City,
			&delivery.Zipcode,
			&delivery.Address,
			&delivery.Phone1,
			&delivery.Phone2,
			&delivery.Cancelled,
			&delivery.Delivered,
			&delivery.DeliveryAttempts,
			&delivery.Driver.FirstName,
			&delivery.Driver.LastName,
		)
		if err != nil {
			return nil, err
		}
		deliveryslice = append(deliveryslice, delivery)
	}
	return deliveryslice, nil
}

func (pdb *postgresDB) ReturnOne(orderNumber int) (*Delivery, error) {
	var delivery = &Delivery{}
	err := pdb.client.QueryRow(retrieveOneDeliveryQuery, orderNumber).Scan(
		&delivery.OrderNumber,
		&delivery.City,
		&delivery.Zipcode,
		&delivery.Address,
		&delivery.Phone1,
		&delivery.Phone2,
		&delivery.Cancelled,
		&delivery.Delivered,
		&delivery.DeliveryAttempts,
		&delivery.Driver.FirstName,
		&delivery.Driver.LastName,
	)
	if err != nil {
		return nil, err
	}
	return delivery, nil
}

func (pdb *postgresDB) Store(d *Delivery) (*Delivery, error) {
	err := pdb.client.QueryRow(storeDeliveryQuery,
		d.City,
		d.Zipcode,
		d.Address,
		d.Phone1,
		d.Phone2,
		d.Cancelled,
		d.Delivered,
		d.DeliveryAttempts,
		d.Driver.FirstName,
		d.Driver.LastName,
	).Scan(&d.OrderNumber)
	if err != nil {
		return nil, err
	}
	return d, nil
}

func (pdb *postgresDB) Change(orderNumber int, del *Delivery) (*Delivery, error) {
	_, err := pdb.client.Exec(changeDeliveryQuery,
		orderNumber,
		del.City,
		del.Zipcode,
		del.Address,
		del.Phone1,
		del.Phone2,
		del.Cancelled,
		del.Delivered,
		del.DeliveryAttempts,
		del.Driver.FirstName,
		del.Driver.LastName,
	)
	if err != nil {
		return nil, err
	}
	return pdb.ReturnOne(orderNumber)
}

func (pdb *postgresDB) Remove(orderNumber int) error {
	_, err := pdb.client.Exec(removeDeliveryQuery, orderNumber)
	if err != nil {
		return err
	}
	return nil
}

Just pass the order number and execute the query. Simple enough.

Run last test

With the code completed, run postgres using the docker command:

`1`	`docker run -d --rm --name testpostgres -p 5432:5432 -e POSTGRES_PASSWORD=Apasswd -e POSTGRES_USER=tester postgres:latest`

and then run the following commands to test all functionality:

curl http://localhost:8000/api/v1/deliveries
curl -X POST -d '{"ordernumber":2,"city":"There","zipcode":"1701","address":"Office","phone1":"6932728091","cancelled":false,"delivered":true,"deliveryattempts":1,"deliverydriver":{"firstname":"Lucas","lastname":"Johnson"}}' http://localhost/api/v1/delivery
curl http://localhost:8000/api/v1/deliveries
curl -X UPDATE -d '{"ordernumber":97,"city":"Here","zipcode":"1701","address":"Home Office","phone1":"6932728091","cancelled":false,"delivered":true,"deliveryattempts":1,"deliverydriver":{"firstname":"Lucas","lastname":"Johnson"}}' http://localhost/api/v1/delivery
curl http://localhost:8000/api/v1/deliveries
curl -X DELETE http://localhost/api/v1/delivery/1
curl http://localhost:8000/api/v1/deliveries

Ta-da! We now have a web service backed by a database. It’s sophisticated enough to do the basic CRUD operations and it’s a fairly relatable domain; deliveries.

Conclusion

That was a long one, wasn’t it? It took forever to write, about a month of working on it on and off. Tried my best to give a complete picture and go through the process step by step, let me know how you think that went. Next time I’d like to cover testing but I’m not sure when that part will come out. Until then, I hope you learned something.

The Power of the Library Example

Thu, 06 May 2021 14:25:41 +0300

While considering what kind of example I could use for a service that was supposed to be a tech demo, I settled on a catalog/library of books. I think it’s a powerful example because it includes the basic CRUD operations and also allows for a lot of flexibility and detail of implementation. Below I’ll try to explain the most important ones and maybe help convince you to rethink about it and maybe use it as the next step after the usual todo list one.

The Library example

Let’s start with an explanation of what this example actually is before going into its advantages. It’s pretty simple, you have a library of books and want to perform operations associated with managing a library. Simple concept, simple to implement and uses real-life things that people have experience with and can reason about. Is it really this simple though? In the intro I said it’s supposed to be a step-up from the usual todo list example but so far it doesn’t seem different other than being another object in the place of a todo item. Well, let’s dive a little deeper.

Data Structures

When you start, there is the dilemma of what fields/attributes to give the book. The ISBN stands out as a convenient unique identifier which is great, then there is the title, the number of pages, year of release, format as in hard cover or soft cover or pdf or epub or mobi, edition, categor(y/ies), author, publisher, subtitle, cover art, bookmark for reading progress, description in the back cover, if you own it or not, did you lend it to someone or not, are you lending it from someone or not, if it’s a physical copy where is it located and so on and so forth. I think you get my point, you can get really deep into it if you want or pick some of the basics and go from there. Are they really basics though? Specifically when it comes to who wrote the book, it could be one person or it could be more than one. Especially in technical books it’s not unusual to have 2 or 3 authors so you’re getting into a situation where you have something like the following (silly example but please stay with me):

{
  "books": [
    {
      "ISBN": "0189219181",
      "Title": "hey",
      "Author": {
        "authors": [
          {
            "AuthorID": 1,
            "FirstName": "anguish",
            "MiddleName": "none",
            "LastName": "mental",
            "YearBorn": 404,
            "YearDied": 201,
            "BooksWritten": 5
          },
          {
            "AuthorID": 2,
            "FirstName": "big",
            "MiddleName": "boo",
            "LastName": "ba",
            "BooksWritten": 9543
          },
          {
            "AuthorID": 3,
            "FirstName": "me",
            "MiddleName": "notme",
            "LastName": "notyou",
            "YearBorn": 1999,
            "BooksWritten": 2
          }
        ]
      },
      "Year": 1999,
      "Edition": 1,
      "Publisher": {
        "PublisherID": 3,
        "Name": "urmom",
        "YearStarted": 1237,
        "YearEnded": 2077,
        "BooksPublished": 9001
      },
      "Pages": 100,
      "Category": "tech",
      "PDF": false,
      "Owned": true,
      "Lended": true
    }
  ]
}

Here we see that each author is inside an array which is the value of a key called authors which is the value of a key called Author which is a part of one of the objects in the array called books which is inside the outer JSON object. Fairly complex, right? Additionally, you have to pay attention to this throughout the entire application including saving it and retrieving it from a database, when passing it to other services that might use this data for example a cover art finder or a media aggregator service and when those services process it. Following that, there are logistical considerations about storing an author or publisher twice since the client shouldn’t have to deal with sending the right ID for the publisher or the author(s), checking if a book with the same ISBN exists, potentially handling cases where you bought it again cause the person you gave it to never gave it back and so on. The main takeaway is that you can decide how real-world you want to make the example, add complexity at will and introduce edge cases that defy previously established preconditions such as the ISBN being a unique identifier.

Functions

I touched on this before I believe it warrants repeating. Similar to the data structure you go with, there several functions of differing depth you can implement. The simple CRUD operations might be taken as a baseline but what even if that rule was able to be defied? Do you want to delete the book or simply set a hidden state because you want the library to act as an archive? Should the cases of the author’s year of birth and death be allowed to be set incorrectly (like in the example above with author 1), partially (author 2) or not at all (author 3)? Should the person that the library belongs to be able to mark the books as lended but not necessarily to who or should they be able to record who they lended the books to? Are the books editable after being added or not? Should there be a url for the digital edition visible to anyone or only a signed-in and authenticated person? Do we even want to support multiple people using the same instance of this library service? So many different things to consider and this can be a perfect opportunity to learn to use data while also being the one that defines what that data should look like. You get be the one producing and consuming the data which is not the case with 3rd-party APIs thus giving you the opportunity to figure things out on the way there. And speaking of accessing the data…

Interfacing

I’ve only spoken about server-side stuff this far (unless you count the output of jq -M as a frontend) but let’s round this out with the client-side. I showed some example JSON previously but in the intro there was a mention of a tech demo and you can bet I’m not writing REST for this. The main definition in my case is a protobuf file so I get control of the struct only as far as the proto v3 syntax allows. This presents an interesting challenge where there is limited or no option for annotations so no ORM (gorm for golang) can be used. Since I’m using gRPC with that and also want JSON to be able to be returned, a translation layer was used, grpc-gateway to be specific. With the ability to use JSON to communicate, a cli, desktop, mobile or web client can be made in a variety of languages while not losing the great service-to-service communication that gRPC provides. After having mostly nailed down the data structures I wanted to use and most of the entity-related functions I wanted to implement I decided to take a look at reactjs again. A post about it will come later after I’m done with this project but the example is given to demonstrate that you can pick something that can make an http request and process JSON then run off with it and build something. The way you choose access the library is noteworthy and can become another important part of using this concept as the example in a tutorial.

Conclusion

I believe the book/library example to be a very good one as a second step after a todo list one due to the complexity that it allows to be introduced in the data structures, the functions, the interfaces and APIs as well as the possibilities it opens up. Thank you for reading, I hope you enjoyed it and maybe learned something.

Topackage

Mon, 03 May 2021 12:41:50 +0300

Software I’d like to package for Gentoo

ghostwriter (GH wereturtle/ghostwriter)
tizonia (GH tizonia/tizonia-openmax-li)
x16-emulator (GH commanderx16/x16-emulator)
xcp (GH tarka/xcp)
pyshelf (GH th3r00t/pyShelf)
snake-go (GH tristangoossens/snake-go)
getsum (GH getsumio/getsum)
mangl (GH zigalenarcic/mangl)
telnet-games (GH Milek7/telnet-games)
sftpgo (GH drakkan/sftpgo)
plex-mpv-shim (GH iwalton3/plex-mpv-shim)
fig (GH flipcoder/fig)
xinput-gui (GH IvanFon/xinput-gui)
partyloud (GH realtho/PartyLoud)
havesnippet (GH aither/havesnippet)
jmdns (GH jmdns/jmdns)
nbsdgames (GH untakenstupidnick/nbsdgames)
qfsview (GH blastrock/qfsview)
statsserver (GH BeayemX/StatsServer)
usbswitch (GH ajmar/usbswitch)
ly (GH cylgom/ly)
Cloudflare-DDNS-Update (GH MachineITSvcs/Cloudflare-DDNS-Update)
netauth (GH netauth/netauth)
pam_netauth (GH netauth/pam_netauth)
nsscache (GH netauth/nsscache)
netkeys (GH netauth/netkeys)
peertube (GH Chocobozzz/PeerTube)
vlcj (GH caprica/vlcj)
javampd (GH finnyb/javampd)
kodeshare (git.kageru.moe kageru/kodeshare)
nci-ansible-ui (GH node-ci/nci-ansible-ui)
synapse (GH matrix-org/synapse)
pdd (GH jarun/pdd)
oolite (GH OoliteProject/oolite)
pioneer (GH pioneerspacesim/pioneer)
Clight (GH FedeDP/Clight)
restic (GH restic/restic)
EternalTerminal (GH MisterTea/EternalTerminal)
reddio (GH TheXamlGuy/Reddio)
tuir (GL ajak/tuir)
zfscrypt (GH BenKerry/zfscrypt)
pash (GH dylanaraps/pash)
eagle (GH im-n1/eagle)
streama (GH streamaserver/streama)
send (GH mozilla/send)
syncserver (GH mozilla-services/syncserver)
jellyfin (GH jellyfin/jellyfin)
goldberg_emu (GL Mr_Goldberg/goldberg_emulator)
jexer (GL klamonte/jexer)
teleport (GL jsparber/teleport)
arch-wiki-docs (GH lahwaacz/arch-wiki-docs)
mumble-web (GH Johni0702/mumble-web)
netris (https://git.sr.ht/~tslocum/netris?0.1.2)
girens (GL tijder/girens)
pty4j (GH JetBrains/pty4j)
jediterm (GH traff/jediterm)
virt-lightning (GH virt-lightning/virt-lightning)
fre:ac (GH enzo1982/freac)
freetube (GH FreeTubeApp/FreeTube)
chat (GH tinode/chat)
publii (GH GetPublii/Publii)
mdless (GH ttscoff/mdless)
broot (GH Canop/broot)
mdcat (GH lunaryorn/mdcat)
translate-shell (GH soimort/translate-shell)
epr (GH wustho/epr)
taskell (GH smallhadroncollider/taskell)
imgp (GH jarun/imgp)
bcal (GH jarun/bcal)
supysonic (GH spl0k/supysonic)
lf (GH gokcehan/lf)
pup (GH ericchiang/pup)
hledger (GH simonmichael/hledger)
snipe-it (GH snipe/snipe-it)
zeronet (GH HelloZeroNet/ZeroNet)
grex (GH pemistahl/grex)
freecell (GH nfultz/freecell)
termdown (GH trehn/termdown)
boohy (GH anaseto/boohu) (https://git.tuxfamily.org/boohu/boohu.git/) (https://download.tuxfamily.org/boohu/downloads/)
minidlna (GH NathanaelA/minidlna) (fork with password support)
terminal-typeracer (GL ttyperacer/terminal-typeracer)
terminal-tetris (GH ravi-ojha/Terminal-Tetris)
tetris (GH samtay/tetris)
tint (GH DavidGriffith/tint)
virtscreen (GH kbumsik/VirtScreen)
tedicross (GH TediCross/TediCross)
asciidocfx (GH asciidocfx/AsciidocFX)
ignite (GH weaveworks/ignite)

Software

Mon, 03 May 2021 12:40:25 +0300

Interesting Software

Droning On

Sat, 01 May 2021 18:54:59 +0300

I described previously how the deployment of applications is automated in my homelab but didn’t touch upon how I test my own code. The topic came up when I started trying to simulate a development pipeline who’s first step after committing code is to be run through the continuous integration system. What should that system be though? In this post, I’ll take you through the things I tried first and then why I went with drone in the end.

Choose a Fighter

My quest is to ultimately end up with the skills to set up kubernetes so that it can be deployed to everything from a DIY NAS and a stack of NUCs to a warehouse full of servers backed by distributed storage. I’m obviously more towards the first camp so that’s where my journey is at. With that said, my goal is to have software small and lightweight enough that it can run on the former setup easily so that there can be multiple instances of it in the latter form. The current mental model I have is developer organization with not much more than 3 NUCs + 1 NAS per team without losing the ability for continuous integration and continuous deployment. Therefore the landscape needs to be explored and the obvious titan of the old world is jenkins with many newcomers eager to take its place. The obvious caveats still apply, it must be open-source, able to be self-hosted, support for gitea is needed and being able to run well on kubernetes is a foregone conclusion.

Jenkins

It should need no introduction, most developers are familiar with the trusty crusty old butler software. Jenkins is probably one of the most prominent pieces of CI/CD software that was pushed to the forefront when the push for more automation and increased code quality really happened. Sadly there are a few reasons that annoyed me from the get-go. First, the unlocking procedure requires you to kubectl logs <podname> in order to find the unlock key which I understand but nonetheless find annoying. Second, the Jenkinsfile has its own domain-specific language (DSL for short) that I find unappealing even though you won’t catch me praising YAML’s lack of curly braces. I begrudgingly went though it and wrote a Jenkinsfile to give it a fair chance but it wasn’t a nice experience. Next, a single instance uses too much ram to the point that it can barely fit in my k3s VMs which use 2.5GB of ram each together with other software. This is an issue because I like running at least 2 instances of most applications to be sure that everything works correctly when scaling up. Last but not least, it’s a 2-in-1 solution which I don’t find appealing as far as architecture and since I was planning to find a standalone piece of software to do CI and another one to handle CD, it was a hard sell from the start. As a redeeming quality it has an official kubernetes pipeline plugin which is nice so they’ve at least put thought into this use case. Ultimately it wasn’t the right fit for me so it was time to move on.

Drone

While pondering this I remembered an old post from one of the blogs I read, specifically this one. I said what the hell, why not and started looking its website. Drone is a very nice and simple continuous integration tool. From the start it seemed like it would be a good pick but you never know until you try. I wrote some kubernetes manifests, got angry until I got them to work and a half hour later it was hooked up to my local gitea instance and ready to use. I added a .drone.yml to one of my repos, pushed the change, activated the repo from the drone dashboard, made another change to the source code, pushed it and waited to see what happened. Instantly drone received the webhook event and got to work. The dashboard is pretty simple (and sadly lacks a dark mode) so you just hit the thing you want to look at, click on the build and see it happen. It supports different things running at the same time like databases if you want your tests to include that which can be run either as what drone calls services aka run in the background from the beginning or if you want an ordered dependency you can use detached steps which is especially useful in microservices. My only real complaint aside from the lack of dark mode is the inability to trigger a build manually without pushing but that’s only an issue when you first activate a repository inside drone’s dashboard. All in all, it has been a joy to use, most of the projects I am currently developing have a .drone.yml and get built at least 3 times a day when gitea mirrors the gitlab or github repo they’re mainly hosted at.

Conclusion

This wasn’t as big of a journey as finding a continuous development tool because really I only found the two aforementioned options that had the basics I was looking for and one of them didn’t really fit the bill. I’m not huge into software development, it’s mostly a hobby for me as might be obvious by now, since my main concern is infrastructure-related stuff. With that said it’s useful to have the skills to build an application that can communicate with the kubernets api to automate something or write an ansible module and most of all it allows me to see the developer perspective. I haven’t started looking into SOPS or renovate bot or the k3s upgrade controller or velero but I hope to get the chance to do so in the future. Thank you for reading, I hope you enjoyed it and maybe learned something.

Fluxing My Cluster

Sat, 17 Apr 2021 17:24:22 +0300

It’s no secret that I’m a fan of automation and making life easier (even if I made it harder in the first place). One of the issues I’ve been having in my homelab is dealing with deploying stuff to kubernetes. Initially I wanted to just add a plugin to drone and be done with it. However, that didn’t really pan out which ended up being to my benefit since I discovered flux.

The problem

The cluster in its current state is a little sad because I end up doing too many things manually. Whenever something changes it’s time to bring out ye old kubectl apply -f in a local clone of the repo and that doesn’t spark joy. This was obviously bad form and couldn’t continue. What you also have to keep in mind is that I’ve chosen a 3rd-party storage plugin, democratic-csi and assume it’s the default storageclass in the cluster. This means that on a new cluster it’s required that before anything else, that one is added as a storageclass and set as default and local-path is removed from being a default. After that, the applications can be deployed which includes a mix of stuff I’ve written for the purposes of testing as well as 3rd-party software. Now that the stage has been set, let’s see what I tried

Solutions

Drone CI plugin

Initially the idea was that since I’ve alrady solved this for code using drone for CI (this will be discussed in a later post), I could just use a plugin and be done. However, my experience with writing helm charts is limited and the only official plugin for deploying to kubernetes is for helm charts, not kubernetes manifests. Furthermore, as I mentioned previously, not all software deployed on the cluster is something that I have written so even if there was an official kubernetes manifest plugin it wouldn’t cover 3rd-party software. At this point, I was thinking of finding some all-inclusive CI/CD software that would cover everything I wanted to do. It took no more than 10 seconds of thinking to realize that I was looking for non-modular software and discard that idea.

Standalone Continuous Delivery

This was the obvious choice but the software in this category is plentiful. The main competitors were flux, argo and tekton. Their designs differ quite a bit so there is going to be quite a bit of opinion in the following analysis so just keep that in mind. The few things they had in common were that they were all advertised as cloud-native with support for kubernetes, had a cli and were written in Go.

ArgoCD

I started with argo because one of the people that wrote a really good blog post on using kubernetes at home mentioned that they use it so I thought I’d try it first. The installation process required a massive yaml file that you just curl | kubectl apply -f which I wasn’t a big fan of but alright whatever. It also had a weird unlock procedure like jenkins where you have to find a token generated at runtime to unlock it but that’s okay, just an one-time setup thing. After spending a couple evenings with it, I wasn’t satisfied with the experience. The docs weren’t really good at explaining all the argo custom resources and getting a basic single-pod application running took me more than 2 hours. It was doable but this was going to be something I had to do for every application running on my cluster and the prospect didn’t seem appealing. I’m sure with more time I could maybe become more familiar and eventually warm up to it however first impressions were bad so I decided to move on.

Tekton

Tekton’s marketing was interesting so I took a look at it next. It seemed to be easier to set up and a couple random blog posts seemed to praise it so I downloaded another massive installation yaml which the guide told me to just kubectl apply -f which is okay I guess. I spent a few hours with it and it became clear you basically had to use it both for CI and CD so I put it to the side and moved on.

Flux

So why did I mention this one first but try it last? A few reasons. While researching, the v1 -> v2 development effort was going on and v2 didn’t have a lot of features but v1 was being phased out so I didn’t know if I should spend time learning v1 while v2 was coming along or if it’s worth jumping into alpha/beta stage software to avoid using legacy versions. By the time I did get around to it though v2 was clearly the way forward. Not only was it officially endorsed but among people that self-host, it was already the way to go 6-12 months ago. The docs were pretty clear, the cli was cross-platform and could be installed on a Raspberry Pi (something that argo at the time didn’t have available) and the examples worked. In an evening I had learned basically everything I needed to in order to just drop it in, generate the flux CRDs for my existing kubernetes manifests as well as helm charts and be up and running. The only issue I had, probably just a pet peeve of mine, was that the repo it creates on github is private by default and my goal is to have everything public and also secure (not there yet) but whatever. Flux v2 also mentioned as the gitops toolkit is made up of several different parts that have a specific purpose which is something I certainly appreciate. All in all, I think I’m going to stick with flux but you never know how homelab things will end up.

Conclusion

Flux is a really great project and I’ve enjoyed using it so far. The problem of continuous delivery has been solved for me and I recommend checking out if you’re in a similar position. Next up, I’m going to look at renovate bot for automating image updates, SOPS for secret management (which flux supports and an upgrade controller) to further help automate operations. Thank you for reading, I hope you enjoyed it and maybe learned something.

Go API Part 2

Tue, 06 Apr 2021 02:23:55 +0300

Previous relevant tutorial

The introduction, setup and explanation of basic concepts and operations was covered in part 1 so make sure to read that first.

Intro

In this part we will be developing a full CRUD REST API in Go except without a real database because I realize that it’s too much to go over at once. The rest of what I promised is true though, we’ll use real data that is more than just a basic todo, explore routing options, subrouters and even middleware.

Starting point

The place we start at is a basic main.go file with our structs that represent the models we’ll work with as well as a simple gorilla/mux setup. The first part of CRUD that we will implement is the simplest one, Read. More specifically reading all the entries. We will also create a slice (essentially a list) of pointers for the instances of Delivery that will exist. The HTTP Method we specify that we accept is http.MethodGet since whoever is accessing the API is getting data from us, not sending us anything. In addition, we set the http status code to http.StatusOK in order to communicate that the request went well.

// main.go
package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
)

type DeliveryDriver struct {
	FirstName string `json:"firstname"`
	LastName  string `json:"lastname"`
}

type Delivery struct {
	OrderNumber      int            `json:"ordernumber"`
	City             string         `json:"city"`
	Zipcode          string         `json:"zipcode"`
	Address          string         `json:"address"`
	Phone1           string         `json:"phone1"`
	Phone2           string         `json:"phone2,omitempty"`
	Cancelled        bool           `json:"cancelled"`
	Delivered        bool           `json:"delivered"`
	DeliveryAttempts int            `json:"deliveryattempts"`
	Driver           DeliveryDriver `json:"deliverydriver"`
}

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{}

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(deliveries)
}

func main() {
	// Set up router
	router := mux.NewRouter()
	// Set up subrouter for api version 1
	apiV1 := router.PathPrefix("/api/v1").Subrouter()
	// Set up routes
	apiV1.HandleFunc("/deliveries", GetAllDeliveries).Methods(http.MethodGet)
	// Start http server
	log.Fatal(http.ListenAndServe(":8000", router))
}

As you can see it’s fairly familiar. The subrouter is based on the path so anything starting with /api/v1 is sent to that subrouter and all the HandleFunc only needs to specify the path under the path of the subrouter, in this case /deliveries. We can run this using go run main.go and visit localhost:8000/api/v1/deliveries to see a measly [] which is an empty json array.

Get all

Let’s add some data then, 3 example entries should suffice:

// main.go
package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
)

type DeliveryDriver struct {
	FirstName string `json:"firstname"`
	LastName  string `json:"lastname"`
}

type Delivery struct {
	OrderNumber      int            `json:"ordernumber"`
	City             string         `json:"city"`
	Zipcode          string         `json:"zipcode"`
	Address          string         `json:"address"`
	Phone1           string         `json:"phone1"`
	Phone2           string         `json:"phone2,omitempty"`
	Cancelled        bool           `json:"cancelled"`
	Delivered        bool           `json:"delivered"`
	DeliveryAttempts int            `json:"deliveryattempts"`
	Driver           DeliveryDriver `json:"deliverydriver"`
}

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{
	&Delivery{
		OrderNumber:      1,
		City:             "Here",
		Zipcode:          "52011",
		Address:          "Home",
		Phone1:           "6945123789",
		Phone2:           "2313722903",
		Cancelled: false,
		Delivered:        false,
		DeliveryAttempts: 0,
		Driver: DeliveryDriver{
			FirstName: "Mhtsos",
			LastName:  "Iwannou",
		},
	},
	&Delivery{
		OrderNumber:      2,
		City:             "There",
		Zipcode:          "1701",
		Address:          "Office",
		Phone1:           "6932728091",
		Cancelled: false,
		Delivered:        true,
		DeliveryAttempts: 1,
		Driver: DeliveryDriver{
			FirstName: "Lucas",
			LastName:  "Johnson",
		},
	},
	&Delivery{
		OrderNumber:      3,
		City:             "FarAway",
		Zipcode:          "920639",
		Address:          "Island",
		Phone1:           "6900777123",
		Cancelled: true,
		Delivered:        false,
		DeliveryAttempts: 24,
		Driver: DeliveryDriver{
			FirstName: "Pilotos",
			LastName:  "Aeroplanou",
		},
	},
}

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(deliveries)
}

func main() {
	// Set up router
	router := mux.NewRouter()
	// Set up subrouter for api version 1
	apiV1 := router.PathPrefix("/api/v1").Subrouter()
	// Set up routes
	apiV1.HandleFunc("/deliveries", GetAllDeliveries).Methods(http.MethodGet)
	// Start http server
	log.Fatal(http.ListenAndServe(":8000", router))
}

Now after running go run main.go and going to localhost:8000/api/v1/deliveries we can see a whole lot more and if you’re using firefox you’ll get some pretty sweet formatting too (that’s due to setting Content-Type to application/json, if you delete that line it won’t format it).

Get one

Seeing all the entires is fine and dandy but we might want to see information about only one of them. In contrast to the previous one where we used /api/v1/deliveries which means many (in fact it means all because we didn’t specify a range), here we’re going to do something different. We’ll use /api/v1/delivery/ordernumber (notice that delivery is singular) and we’ll specify the ordernumber since it is unique as seen from the example entries. As for the code within the function, we’ll check the parameters to make sure that the argument is actually a number we can use to find the delivery and if it’s fine we will return the delivery as JSON as well as set the status code to http.StatusOK. However if it’s wrong, we’ll set the status code to http.StatusNotFound and then return an error message.

// main.go
package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
	"strconv"
)

type DeliveryDriver struct {
	FirstName string `json:"firstname"`
	LastName  string `json:"lastname"`
}

type Delivery struct {
	OrderNumber      int            `json:"ordernumber"`
	City             string         `json:"city"`
	Zipcode          string         `json:"zipcode"`
	Address          string         `json:"address"`
	Phone1           string         `json:"phone1"`
	Phone2           string         `json:"phone2,omitempty"`
	Cancelled        bool           `json:"cancelled"`
	Delivered        bool           `json:"delivered"`
	DeliveryAttempts int            `json:"deliveryattempts"`
	Driver           DeliveryDriver `json:"deliverydriver"`
}

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{
	&Delivery{
		OrderNumber:      1,
		City:             "Here",
		Zipcode:          "52011",
		Address:          "Home",
		Phone1:           "6945123789",
		Phone2:           "2313722903",
		Cancelled: false,
		Delivered:        false,
		DeliveryAttempts: 0,
		Driver: DeliveryDriver{
			FirstName: "Mhtsos",
			LastName:  "Iwannou",
		},
	},
	&Delivery{
		OrderNumber:      2,
		City:             "There",
		Zipcode:          "1701",
		Address:          "Office",
		Phone1:           "6932728091",
		Cancelled: false,
		Delivered:        true,
		DeliveryAttempts: 1,
		Driver: DeliveryDriver{
			FirstName: "Lucas",
			LastName:  "Johnson",
		},
	},
	&Delivery{
		OrderNumber:      3,
		City:             "FarAway",
		Zipcode:          "920639",
		Address:          "Island",
		Phone1:           "6900777123",
		Cancelled: true,
		Delivered:        false,
		DeliveryAttempts: 24,
		Driver: DeliveryDriver{
			FirstName: "Pilotos",
			LastName:  "Aeroplanou",
		},
	},
}

// Constant for Bad Request
const BadReq string = `{"error": "bad request"}`
// Constant for Not Found
const NotFound string = `{"error": "not found"}`

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(deliveries)
}

// Read a specific delivery
func GetDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

func main() {
	// Set up router
	router := mux.NewRouter()
	// Set up subrouter for api version 1
	apiV1 := router.PathPrefix("/api/v1").Subrouter()
	// Set up routes
	apiV1.HandleFunc("/deliveries", GetAllDeliveries).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery/{ordernumber}", GetDelivery).Methods(http.MethodGet)
	// Start http server
	log.Fatal(http.ListenAndServe(":8000", router))
}

I went ahead and made the errors const variables in the global scope so we can use them anywhere we want without having to repeat ourselves, this is not necessary but it makes life a bit easier. Let’s look at a couple things that seem a bit odd. First what’s that mux.Vars(r)? That’s from gorilla/mux and one of the reasons we are using it. It takes a request as an argument and can tell us a lot of stuff about that request. The variable params is a type map[string]string which means that the key is a string and the value is a string. If you’ve used dictionaries in Python I think it’s pretty similar. We give it the ordernumber key which is what we specified in apiV1.HandleFunc("/delivery/{ordernumber}", GetDelivery).Methods(http.MethodGet) and then use the strconv library to convert it from string to an integer. Second, the for loop looks a bit weird, what’s up with that range deliveries? Using range we can iterate over something, in this case a slice (essentially a list). This returns two variables, in this case one that is the key/index which we are ignoring by using _ and the other one is the value d which is a single delivery from the list of deliveries we set up. With all that out of the way, time to test it. Run the server with go run main.go and visit localhost:8000/api/v1/deliveries/2 to see the information about the second delivery returned. Pretty good, right?

Add one

After we’re done with the Read part of CRUD, let’s move on to Create. Here we will add a delivery after it is sent to us as JSON. I will use cURL for this purpose but you can use something else if you want. Let’s look at the code first though. In our addDelivery function we will create an empty delivery item, d, that will later on receive the JSON data. Then we’ll see what the next order number should be according to the length of our list and use that instead of whatever the user has provided, it is up to us to decide where to fit the delivery. After than, we unpack the contents of the request body into d and check for errors, if there are any, we will handle it like previously since the sender made a mistake. If the ordernumber in their JSON is empty, we don’t mind since we’ll override it before saving it anyway. Following that, we will use append to add the delivery to the list and then set the HTTP status code to http.StatusCreated. Finally we will return the item that we actually inserted into our list of deliveries so the sender knows what was actually saved.

// main.go
package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
	"strconv"
)

type DeliveryDriver struct {
	FirstName string `json:"firstname"`
	LastName  string `json:"lastname"`
}

type Delivery struct {
	OrderNumber      int            `json:"ordernumber"`
	City             string         `json:"city"`
	Zipcode          string         `json:"zipcode"`
	Address          string         `json:"address"`
	Phone1           string         `json:"phone1"`
	Phone2           string         `json:"phone2,omitempty"`
	Cancelled        bool           `json:"cancelled"`
	Delivered        bool           `json:"delivered"`
	DeliveryAttempts int            `json:"deliveryattempts"`
	Driver           DeliveryDriver `json:"deliverydriver"`
}

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{
	&Delivery{
		OrderNumber:      1,
		City:             "Here",
		Zipcode:          "52011",
		Address:          "Home",
		Phone1:           "6945123789",
		Phone2:           "2313722903",
		Cancelled: false,
		Delivered:        false,
		DeliveryAttempts: 0,
		Driver: DeliveryDriver{
			FirstName: "Mhtsos",
			LastName:  "Iwannou",
		},
	},
	&Delivery{
		OrderNumber:      2,
		City:             "There",
		Zipcode:          "1701",
		Address:          "Office",
		Phone1:           "6932728091",
		Cancelled: false,
		Delivered:        true,
		DeliveryAttempts: 1,
		Driver: DeliveryDriver{
			FirstName: "Lucas",
			LastName:  "Johnson",
		},
	},
	&Delivery{
		OrderNumber:      3,
		City:             "FarAway",
		Zipcode:          "920639",
		Address:          "Island",
		Phone1:           "6900777123",
		Cancelled: true,
		Delivered:        false,
		DeliveryAttempts: 24,
		Driver: DeliveryDriver{
			FirstName: "Pilotos",
			LastName:  "Aeroplanou",
		},
	},
}

// Constant for Bad Request
const BadReq string = `{"error": "bad request"}`
// Constant for Not Found
const NotFound string = `{"error": "not found"}`

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(deliveries)
}

// Read a specific delivery
func GetDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

// Create a new delivery
func AddDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	d := &Delivery{}
	// because deliveries start from 1
	orderNum := int(len(deliveries)) + 1
	err := json.NewDecoder(r.Body).Decode(d)
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	// Override whatever the sender had sent
	d.OrderNumber = orderNum
	// Append the delivery to deliveries list
	deliveries = append(deliveries, d)
	w.WriteHeader(http.StatusCreated)
	// Send back the delivery that was saved
	json.NewEncoder(w).Encode(d)
	return
}

func main() {
	// Set up router
	router := mux.NewRouter()
	// Set up subrouter for api version 1
	apiV1 := router.PathPrefix("/api/v1").Subrouter()
	// Set up routes
	apiV1.HandleFunc("/deliveries", GetAllDeliveries).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery/{ordernumber}", GetDelivery).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery", AddDelivery).Methods(http.MethodPost)
	// Start http server
	log.Fatal(http.ListenAndServe(":8000", router))
}

We’ve also added another route that is used when the HTTP method is POST which calls the AddDelivery function we just created. In order to test it out, I made this delivery.json file that you can also use. This is what it contains:

{
  "city": "ExperimentValley",
  "zipcode": "0000",
  "address": "REDACTED",
  "phone1": "6900999111",
  "cancelled": false,
  "delivered": false,
  "deliveryattempts": 3,
  "deliverydriver": {
    "firstname": "Mike",
    "lastname": "REDACTED"
  }
}

I saved it in my home directory with the name delivery.json and from the same place, after starting the server with go run main.go I ran:

`1`	`curl -H "Content-Type: application/json" -X POST --data-binary '@delivery.json' http://localhost:8000/api/v1/delivery`

to send it over to the server which then returned

{"ordernumber":4,"city":"ExperimentValley","zipcode":"0000","address":"REDACTED","phone1":"6900999111","cancelled":false,"delivered":false,"deliveryattempts":3,"deliverydriver":{"firstname":"Mike","lastname":"REDACTED"}}

which as we can see has ordernumber set to 4 even though I didn’t explicitly set it. Feel free to try setting ordernumber to 27 or something inside delivery.json and see what you get.

Upd8 1

Next up we’re going to update an entry. The appropriate HTTP method for this is PUT and the status code is http.StatusAccepted or http.StatusOK and I prefer the first one so that’s what I’ll use. The UpdateDelivery function we’re going to make has elements we saw previously in GetDelivery as well as AddDelivery. Not much of this is new so let’s see how we get things done.

// main.go
package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
	"strconv"
)

type DeliveryDriver struct {
	FirstName string `json:"firstname"`
	LastName  string `json:"lastname"`
}

type Delivery struct {
	OrderNumber      int            `json:"ordernumber"`
	City             string         `json:"city"`
	Zipcode          string         `json:"zipcode"`
	Address          string         `json:"address"`
	Phone1           string         `json:"phone1"`
	Phone2           string         `json:"phone2,omitempty"`
	Cancelled        bool           `json:"cancelled"`
	Delivered        bool           `json:"delivered"`
	DeliveryAttempts int            `json:"deliveryattempts"`
	Driver           DeliveryDriver `json:"deliverydriver"`
}

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{
	&Delivery{
		OrderNumber:      1,
		City:             "Here",
		Zipcode:          "52011",
		Address:          "Home",
		Phone1:           "6945123789",
		Phone2:           "2313722903",
		Cancelled: false,
		Delivered:        false,
		DeliveryAttempts: 0,
		Driver: DeliveryDriver{
			FirstName: "Mhtsos",
			LastName:  "Iwannou",
		},
	},
	&Delivery{
		OrderNumber:      2,
		City:             "There",
		Zipcode:          "1701",
		Address:          "Office",
		Phone1:           "6932728091",
		Cancelled: false,
		Delivered:        true,
		DeliveryAttempts: 1,
		Driver: DeliveryDriver{
			FirstName: "Lucas",
			LastName:  "Johnson",
		},
	},
	&Delivery{
		OrderNumber:      3,
		City:             "FarAway",
		Zipcode:          "920639",
		Address:          "Island",
		Phone1:           "6900777123",
		Cancelled: true,
		Delivered:        false,
		DeliveryAttempts: 24,
		Driver: DeliveryDriver{
			FirstName: "Pilotos",
			LastName:  "Aeroplanou",
		},
	},
}

// Constant for Bad Request
const BadReq string = `{"error": "bad request"}`
// Constant for Not Found
const NotFound string = `{"error": "not found"}`

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(deliveries)
}

// Read a specific delivery
func GetDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

// Create a new delivery
func AddDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	d := &Delivery{}
	// because deliveries start from 1
	orderNum := int(len(deliveries)) + 1
	err := json.NewDecoder(r.Body).Decode(d)
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	// Override whatever the sender had sent
	d.OrderNumber = orderNum
	// Append the delivery to deliveries list
	deliveries = append(deliveries, d)
	w.WriteHeader(http.StatusCreated)
	// Send back the delivery that was saved
	json.NewEncoder(w).Encode(d)
	return
}

// Update an existing delivery
func UpdateDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			err := json.NewDecoder(r.Body).Decode(d)
			if err != nil {
				w.WriteHeader(http.StatusBadRequest)
				w.Write([]byte(BadReq))
				return
			}
			d.OrderNumber = orderNum
			w.WriteHeader(http.StatusAccepted)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

func main() {
	// Set up router
	router := mux.NewRouter()
	// Set up subrouter for api version 1
	apiV1 := router.PathPrefix("/api/v1").Subrouter()
	// Set up routes
	apiV1.HandleFunc("/deliveries", GetAllDeliveries).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery/{ordernumber}", GetDelivery).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery", AddDelivery).Methods(http.MethodPost)
	apiV1.HandleFunc("/delivery/{ordernumber}", UpdateDelivery).Methods(http.MethodPut)
	// Start http server
	log.Fatal(http.ListenAndServe(":8000", router))
}

As we can see, nothing special, same stuff we’ve done before. Check order number, return error bad request if it’s bad. Go through the deliveries to find the one, if it doesn’t exist return not found. If it’s found, try to unpack the sender’s json to the existing delivery and if that fails return bad request otherwise save changes and return the delivery that was saved in our list. Note that we also override the value that the sender gave to keep our deliveries the way we want them and not have duplicates of what is supposed to be a unique identifier. Also we added the route on the bottom to make it usable. Similar to what we did to test it earlier, start the server with go run main.go and then run the following to change the data of delivery number 2:

`1`	`curl -H "Content-Type: application/json" -X PUT --data-binary '@delivery.json' http://localhost:8000/api/v1/delivery/2`

which returns:

{"ordernumber":2,"city":"ExperimentValley","zipcode":"0000","address":"REDACTED","phone1":"6900999111","cancelled":false,"delivered":false,"deliveryattempts":3,"deliverydriver":{"firstname":"Mike","lastname":"REDACTED"}}

and we can tell from ordernumber that it saved it there. To make sure, we can also run

`1`	`curl http://localhost:8000/api/v1/deliveries`

to get a list of all deliveries.

Delete one

I know that Delete is left and we will implement it but we’re not actually going to delete anything. This is an important point to make, we have to consider what the application is doing. If we want to have a history of deliveries, we don’t want any of them to be lost. What we’re going to do instead is set the Cancelled to true. Let’s think about why for a bit. If the order is successfully delivered, we use the UpdateDelivery method to set it and it’s all good. So it’s the job of DeleteDelivery to handle deliveries that need to be invalidated in some other way, not completed deliveries. The only reason I can think of that you’d want to “delete” a delivery is if the order being cancelled so that’s what we’ll do. We do the normal check for the path parameter, if we find a delivery with that number we set it to cancelled and if not we return error not found. Here is the code:

// main.go
package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
	"strconv"
)

type DeliveryDriver struct {
	FirstName string `json:"firstname"`
	LastName  string `json:"lastname"`
}

type Delivery struct {
	OrderNumber      int            `json:"ordernumber"`
	City             string         `json:"city"`
	Zipcode          string         `json:"zipcode"`
	Address          string         `json:"address"`
	Phone1           string         `json:"phone1"`
	Phone2           string         `json:"phone2,omitempty"`
	Cancelled        bool           `json:"cancelled"`
	Delivered        bool           `json:"delivered"`
	DeliveryAttempts int            `json:"deliveryattempts"`
	Driver           DeliveryDriver `json:"deliverydriver"`
}

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{
	&Delivery{
		OrderNumber:      1,
		City:             "Here",
		Zipcode:          "52011",
		Address:          "Home",
		Phone1:           "6945123789",
		Phone2:           "2313722903",
		Cancelled: false,
		Delivered:        false,
		DeliveryAttempts: 0,
		Driver: DeliveryDriver{
			FirstName: "Mhtsos",
			LastName:  "Iwannou",
		},
	},
	&Delivery{
		OrderNumber:      2,
		City:             "There",
		Zipcode:          "1701",
		Address:          "Office",
		Phone1:           "6932728091",
		Cancelled: false,
		Delivered:        true,
		DeliveryAttempts: 1,
		Driver: DeliveryDriver{
			FirstName: "Lucas",
			LastName:  "Johnson",
		},
	},
	&Delivery{
		OrderNumber:      3,
		City:             "FarAway",
		Zipcode:          "920639",
		Address:          "Island",
		Phone1:           "6900777123",
		Cancelled: true,
		Delivered:        false,
		DeliveryAttempts: 24,
		Driver: DeliveryDriver{
			FirstName: "Pilotos",
			LastName:  "Aeroplanou",
		},
	},
}

// Constant for Bad Request
const BadReq string = `{"error": "bad request"}`
// Constant for Not Found
const NotFound string = `{"error": "not found"}`

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(deliveries)
}

// Read a specific delivery
func GetDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

// Create a new delivery
func AddDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	d := &Delivery{}
	// because deliveries start from 1
	orderNum := int(len(deliveries)) + 1
	err := json.NewDecoder(r.Body).Decode(d)
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	// Override whatever the sender had sent
	d.OrderNumber = orderNum
	// Append the delivery to deliveries list
	deliveries = append(deliveries, d)
	w.WriteHeader(http.StatusCreated)
	// Send back the delivery that was saved
	json.NewEncoder(w).Encode(d)
	return
}

// Update an existing delivery
func UpdateDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			err := json.NewDecoder(r.Body).Decode(d)
			if err != nil {
				w.WriteHeader(http.StatusBadRequest)
				w.Write([]byte(BadReq))
				return
			}
			d.OrderNumber = orderNum
			w.WriteHeader(http.StatusAccepted)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

// Delete a delivery (not really)
func DeleteDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			d.Cancelled = true
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

func main() {
	// Set up router
	router := mux.NewRouter()
	// Set up subrouter for api version 1
	apiV1 := router.PathPrefix("/api/v1").Subrouter()
	// Set up routes
	apiV1.HandleFunc("/deliveries", GetAllDeliveries).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery/{ordernumber}", GetDelivery).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery", AddDelivery).Methods(http.MethodPost)
	apiV1.HandleFunc("/delivery/{ordernumber}", UpdateDelivery).Methods(http.MethodPut)
	apiV1.HandleFunc("/delivery/{ordernumber}", DeleteDelivery).Methods(http.MethodDelete)
	// Start http server
	log.Fatal(http.ListenAndServe(":8000", router))
}

First we start the server with go run main.go and then to test our newly added functionality, in my case with cURL, we can run:

`1`	`curl -X DELETE http://localhost:8000/api/v1/delivery/1`

and get back:

{"ordernumber":1,"city":"Here","zipcode":"52011","address":"Home","phone1":"6945123789","phone2":"2313722903","cancelled":true,"delivered":false,"deliveryattempts":0,"deliverydriver":{"firstname":"Mhtsos","lastname":"Iwannou"}}

Conclusion

That’s it for this part then. You now have a fully functional REST API written in Go that you can play around with. There are a few more things that you can do with gorilla/mux and the code can be made a little bit nicer so if you’re interested in that, check out my completed version. Thank you for reading and I hope you enjoyed.

REST Is Over

Sun, 04 Apr 2021 17:33:02 +0300

Web APIs are terrible

After watching a talk a few months back about REST vs GraphQL and the history of APIs, I agreed with the speaker that the API is supposed to be a way of communication, a language of sorts. In programming languages this codified way of communication is enforced by the compiler or the interpreter or the runtime. If you write invalid code in the programming language of your choosing, it won’t work. Something or other will complain or break and prevent you from miscommunicating. This got me wondering APIs especially web ones are so bad.

RESTing comfortably

With the introduction of what we call REST nowadays, basically JSON over HTTP, it’s all too common for application to use this mode of communication. How this is enforced in practice is very variable though. From different path schemes, to the same HTTP verbs used for different purposes, to (invariably bad) ways to communicate from machine to machine what the API is, it’s all up in the air. Most fairly complex things that have a REST API, don’t even expect you to use them by directly interacting with it but instead use client libraries. This has been going on for about a decade or two and it seems things are changing. Some thought that GraphQL would be the end of that and it’s a pretty good step up but adoption is nowhere near as high. In general REST is the assumed default way to interact with something programmatically. Should it be though? Is GraphQL the best replacement we’ve got? I don’t think so.

gRPCee what I did there

A somewhat recent thing called gRPC started picking up steam. Talks about it, articles about it, large software projects using it but not really prominent on the web. With this nifty little thing after writing a file that describes what your program does, you can generate some ready to use code, write the logic and grpc.Serve() to the moon. Even better, anyone wanting to write a client in any of the many supported languages can generate the boilerplate for a client from the same definition file and start calling the functions of the server really easily. I’m being a bit simplistic for the sake of brevity but it’s really cool, trust me. It’s also really fast. Alright, it’s better than freshly baked bread apparently so why is it not used on the web? Cause it’s only HTTP/2 and the support for that is pretty limited. Fear not though, we’re going to circle back to our crusty old frenemy, REST.

Support for REST is widespread and there is a webdev in every city block ready to POST some JSON to your endpoint. So we can’t immediately get rid of the kludgy wart but what about giving pixel pushers some REST while service to service communication is improved? You see, since definitions exist for what the service sends and receives, the types of all the things being passed around, it’s quite easy to use some simple annotations to auto-generate a REST API. This doesn’t 100% fix the problems that exist but it does smooth over some parts of it. Okay so that’s it? Not just that, since the API is standardized we can even auto-generate documentation for it! OpenAPI (formerly Swagger) docs can be easily generated and served to potential downstream users of the API. Brilliant I think.

Conclusions

Over the past few months I’ve been reading a lot about gRPC, grpc-gateway, different ideas and implementations, trying out quite a few of them but I’ve yet to see one that makes me super excited and go “Aha! this is The One” or whatever. I could do a link dump of about 100-150 tabs that I have open but that would be tiring and essentially do nothing to help. Currently I’ve been trying to decide among 2-3 options for how a service should be structured and such but I’m still testing out prototypes, haven’t reached any conclusions. However, I hope this was a semi-educational read and got you a bit interested in the ecosystem. That’s all for now, I hope you enjoyed reading.

pardon me

~~check out the tutorials section for my Go REST API Part 2 coming soon~~ sorry, I had to :^)

Go API Part 1

Thu, 01 Apr 2021 06:39:47 +0300

Intro

This is going to be a simple tutorial about creating a backend web service that can be accessed over a REST API. The application is going to be able to do something very basic just to make everyone familiar with the basics. There will eventually be a part 2 where we will make an application that performs the basic CRUD(Create Read Update Delete) actions, it will include real-life applicable data and actions as well as a database for storing information. Let’s start then.

Getting set up

I’m not going to cover how to set up the development environment but you can look at these links:

Greeting the world

First off, in a file called main.go we declare the package (important note here, if the package is not main then there is no ability to use go run on it), add a couple basic imports, write the main function and make it print something:

package main

import (
	"fmt"
)

func main() {
	fmt.Println("ordering facilities online")
}

And that’s our hello world right there. Use go run main.go to run the program.

Webbing it up

Alright now that that’s done, let’s start messing with the web capabilities. First we’ll select a port depending on if the PORT environment variable has been set or not and then we’ll write a simple endpoint that returns the current year. While we’re at it I’ll switch out fmt for log because I like the output better:

package main

import (
	"log"
	"net/http"
	"os"
	"strconv"
	"time"
)

func getCurrentYear(w http.ResponseWriter, r *http.Request) {
	currentTime := time.Now()
	year := currentTime.Year()
	yearString := strconv.Itoa(year)
	w.Header().Set("Content-Type", "text/plain")
	w.WriteHeader(http.StatusOK)
	w.Write([]byte("current year is: "+yearString))
	return
}

func main() {
	log.Println("ordering facilities online")
	port := os.Getenv("PORT")
	if port == "" {
		port = "8080"
	}
	http.HandleFunc("/year", getCurrentYear)
	log.Fatal(http.ListenAndServe(":"+port, nil))
}

A couple things to comment on. First off, := means that you don’t have to manually declare the type of the variable, it will be figured out automatically by the compiler. Second, the signature of the getCurrentYear function. The arguments might look weird: w http.ResponseWriter, r *http.Request but those two variables are passed to the function by http.HandleFunc. ResponseWriter w allows us to work with what we’ll return and Request r gives us info about the request. Let’s get into the getCurrentYear function. The first thing it does is get the year which is of type int and then using string converter (strconv) to make it a string in order to be able to be added together with our message in the end. After creating yearString we se the content type so the client (in this case the browser) knows what we’re sending and set the status to OK (code 200) so the client knows how things went. The responsewriter writes a slice (indicated by []) of bytes to the client which in this case is the message current year is: along with the year as calculated previously. Inside the main function we have http.HandleFunc("/year", getCurrentYear) which says that if you visit /year then the function getCurrentYear should be run. This function is also called a handler because it handles the request. Lastly, we run http.ListenAndServe(":"+port, nil) inside log.Fatal because it returns an error if it exists so if any error with starting the http server happens, we’ll see it and the program won’t run. To test all of this, run go run main.go and then open a browser, visit localhost:8080/year and see the magic.

JSONify our Year

To give a simple introduction to data structures and the JSON format we’ll work with. First we’ll make a struct which will be our custom type for representing the year with some annotations. We’ll also import encoding/json to translate the struct to json based on the annotations and return it to the client. In addition, we’ll adjust some code accordingly.

package main

import (
	"encoding/json"
	"log"
	"net/http"
	"os"
	"time"
)

type MyYear struct {
	CurrentYear int    `json:"currentyear"`
	Good        bool   `json:"good,omitempty"`
	Comment     string `json:"comment,omitempty"`
}

func getCurrentYear(w http.ResponseWriter, r *http.Request) {
	currentTime := time.Now()
	year := currentTime.Year()
	var goodYear bool
	var comment string
	if (year == 2020 || year == 2021) {
		goodYear = false
		comment = "big oof"
	}
	myCurrentYear := MyYear{
		CurrentYear: year,
		Good: goodYear,
		Comment: comment,
	}
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(myCurrentYear)
	return
}

func main() {
	log.Println("ordering facilities online")
	port := os.Getenv("PORT")
	if port == "" {
		port = "8080"
	}
	http.HandleFunc("/year", getCurrentYear)
	log.Fatal(http.ListenAndServe(":"+port, nil))
}

Alright so there is a lot to go over. First off, the struct MyYear is our custom type. It has 3 fields, CurrentYear to store the year as an integer, Good to store a boolean about the year being good or not and Comment for the year. The things next to each field in the struct specify the name that the json field will have as well as if it should be printed when the field is empty (specified by omitempty). The content type was changed to application/json since that is what we’re returning. In the code above, myCurrentYear is created when the request is sent according to some conditions. There is a little if statement that sets a default value for Good and Comment if the year is 2020 or 2021. The variables goodYear and comment are declared without a default value and if we hadn’t used them the compiler would error out but we did so that’s not an issue just something to keep in mind. All of the translation of the struct to json is done automatically, we just created a new json encoder that encodes into the responsewriter aka to the client and then encoded our struct. If you run go run main.go now and then visit localhost:8080/year you will see that now the response is in json, exactly what we wanted.

Adding a dependency

Eventually we will want to be able to do a GET request on something like example.com/api/v1/orders/521 and get back the details of the 521st order in JSON format. This requires some routing as well as reading path parameters, in this case that parameter is 521. For this purpose we’ll use a request multiplexer, specifically gorilla/mux. To add it to our system we can just do go get github.com/gorilla/mux and then add it as a dependency as shown below:

package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
	"os"
	"time"
)

type MyYear struct {
	CurrentYear int    `json:"currentyear"`
	Good        bool   `json:"good,omitempty"`
	Comment     string `json:"comment,omitempty"`
}

func getCurrentYear(w http.ResponseWriter, r *http.Request) {
	currentTime := time.Now()
	year := currentTime.Year()
	var goodYear bool
	var comment string
	if (year == 2020 || year == 2021) {
		goodYear = false
		comment = "big oof"
	}
	myCurrentYear := MyYear{
		CurrentYear: year,
		Good: goodYear,
		Comment: comment,
	}
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(myCurrentYear)
	return
}

func main() {
	log.Println("ordering facilities online")
	port := os.Getenv("PORT")
	if port == "" {
		port = "8080"
	}
	router := mux.NewRouter()
	router.HandleFunc("/year", getCurrentYear)
	log.Fatal(http.ListenAndServe(":"+port, router))
}

We’ve added it, notice how http.ListenAndServe has a second argument now instead of nil. Also instead of http.HandleFunc we have router.HandleFunc since the handler is no longer the default one from net/http. Now if you visit localhost:8080/year you’ll notice that json is returned just like before. I didn’t comment on it previously but did you notice the lack of good in the json response? Why is that? We explicitly set a value so what’s the deal there? In the case of a bool variable, false is considered to be the “empty” state of the variable so it’s not shown. All we need to do to fix it is remove the omitempty from its annotation (as well as the comma) and we’ll get the result we want. Here is the finished code then:

package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
	"os"
	"time"
)

type MyYear struct {
	CurrentYear int    `json:"currentyear"`
	Good        bool   `json:"good"`
	Comment     string `json:"comment,omitempty"`
}

func getCurrentYear(w http.ResponseWriter, r *http.Request) {
	currentTime := time.Now()
	year := currentTime.Year()
	var goodYear bool
	var comment string
	if (year == 2020 || year == 2021) {
		goodYear = false
		comment = "big oof"
	}
	myCurrentYear := MyYear{
		CurrentYear: year,
		Good: goodYear,
		Comment: comment,
	}
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(myCurrentYear)
	return
}

func main() {
	log.Println("ordering facilities online")
	port := os.Getenv("PORT")
	if port == "" {
		port = "8080"
	}
	router := mux.NewRouter()
	router.HandleFunc("/year", getCurrentYear)
	log.Fatal(http.ListenAndServe(":"+port, router))
}

Adding gorilla/mux didn’t do a lot, we have added no extra functionality to our program but added a dependency which is generally not good. The next part will focus more on the awesome things we can do with gorilla/mux related to paths, subrouters, regular expressions and matching HTTP verbs.

Conclusion

This was a pretty heavy introduction to Go with an emphasis on using it for web services. Feel free to run the examples, tweak them, add to them and become familiar with documentation. Next time we’ll be getting into full CRUD operations, databases and probably a Dockerfile too. Thank you for reading, I hope you enjoyed and maybe learned something.

Golang Is Fucking Rad

Thu, 18 Mar 2021 21:21:15 +0200

Radical Gopherism

I’ve been using Go for the better part of a year at this point and I’m loving most things about it. You know the typical stuff, types, compiled, fast, portable, easy to cross-compile and the rest of the goodness. However that’s not what made me stick with it although those things helped. The main things I liked were mostly that it was in many ways not like Java. There is no need for an IDE, the language is not endlessly complex, I can read other people’s code right away. Don’t forget the compiler-enforced formatting, easy static compilation allowing for tiny container images. The standard library is also phenomenal if frameworks are not your cup of tea it’s very possible to avoid them in a lot of cases. And the new file embedding functionality? Amazing, my site is now a portable self-contained binary. And the adoption? Practically any project I’m interested in just so happens to be written in it! Most things I use, kubernetes, docker, terraform, vault, gitea, droneci, fluxcd, minio and more I’m probably forgetting.

Reality

Time to stop drooling though. While I wasn’t inaccurate or misrepresentative above, I want to look at pros-cons for a moment.

Dank Reality

The main thing I do is write backend web microservices and small operating system utilities. As mentioned previously, the stdlib is amazing and quite a few things I’ve written use that almost exclusively. If web frameworks are needed, there are plenty from piece-by-piece like gorilla, go-kit(depends if you count it) or go-chi, to all-encompassing gargantuan ones like go-micro (ironic name, I know) to middleground solutions like gin-gonic. I can find client libraries for everything I want be it any sort of api, databases, application messaging systems and recently I found out I can even run ML models generated by keras in python from go. I didn’t harp too much on this before but the static compilation is sweet. Using docker multi-stage builds and starting from the 150MB fat golang image with devtools in it but then being able to run the resulting binary in a run stage based on scratch and have a 3MB production image is incredible. I use it more and more every day and basically haven’t looked to shell scripting for anything except heavy string manipulation but even there I’m starting to move as see in gopasswd.

Sad Reality

This site is my doomer cabin in the woods so the fun stops here. Also lists are scary so let’s go

language package manager: having maintained gentoo ebuilds, I’ll never not hate language package managers
not very competitive alternative compiler: gcc-go produces slower artifacts and I don’t want a dotnet situation
google is very heavily involved: the pkg.go.dev shit just showcased it but I’m not comfortable with google being involved and yes this goes for k8s as well
grpc support: google came up with grpc and is too involved in go but you need a 3rd party plugin for protoc to output go which is stupid
dev2prod: translating from local to remote dependencies and vice versa is handled poorly
modules: kinda weird, I’ve come across too many cases where rm go.mod go.sum && go mod init && go mod tidy was needed
generics: don’t want generics but everyone is begging and the advantage of small and understandable might be lost in the future

Conclusion

Good language, fits what I do, like using it, will continue using it, wish some stuff was improved or fixed. That was all for this post, I hope you enjoyed.

Reinstalling Gentoo

Wed, 10 Mar 2021 00:54:04 +0200

Why

This gentoo install has lasted a good long couple of years. There is nothing really wrong with it, the SSD it’s installed on is a bit full and it’s a good opportunity to make a change. However I’m looking to shake things up a bit. The big change is installing on ZFS this time. It’s fairly well-known that I’m a big fan of ZFS, my storage server has been using it in one form or another sinc forever and all of my home infrastructure directly or indirectly depends on it. However it is not the only change being made, executing the install using ansible will be a good improvement too. One of the issues with gentoo, to people that have been using it for a while, is the seeming non-repeatability of the install. A script can get really close however certain steps are not as easy to get right and everything is highly procedural instead of declarative or idempotent way. The other changes will include moving to a system with as few, as I view them, harmful or unnecessary componenets. Stuff like dbus, eudev, elogind, pulseaudio, systray and a few other things that I’m in the process of nailing down will be (hopefully) gone.

How

Ansible

First off, the ansible playbook. There are a few different ones floating around but are outdated or don’t fit my needs. Of course that doesn’t mean I can’t examine the pre-existing solutions, I will absolutely take inspiration from solutions that smarter people have come up with.

ZFS

The most important decision will be the layout. I know I’ll be using docker and libvirt so adjustments to /var/lib are needed. Also, the system needs to be bootable and usable to some extent even if the ZFS kernel module doesn’t get loaded. There are 2 different solutions to this, either go with EXT4/XFS or just build ZFS into the kernel. I’m still undecided about the route I’ll go with but remain open to experimenting and seeing which one is better.

Cruft removal

Several components are impacted here. The most interesting one I’m looking forward to installing is libudev-zero to seemingly satisfy any dependencies that require (e)udev while in reality using busybox mdev. This gets rid of some dependency problems with libinput/evdev and anything else that has a hard dependency on a single device manager implementation without source code patches to those projects. It also give me the ability to use input classes in xorg instead of hardcoding device paths which is something that was somewhat annoying last time I tried switching away from eudev. As for the rest, I’m testing out what can be effortlessly removed and what requires significant changes to one thing or another on my laptop. We’ll see what I end up with, I’ll update this post after it’s up and running.

Conclusion

I’ve enjoyed my gentoo system for multiple years at this point but I want to make things better so I’ll be reinstalling and starting from a clean slate. Switching the filesystem to the one my storage server relies on will be pretty sweet (I might also look into some pre or post package install hooks). The installation process, and hopefully the full system setup, will be described using ansible to make it more easily repeatable And finally, any piece of software I don’t like or use will be removed or replaced and incompatibilites fixed along the way. I look forward to a better gentoo system in 2021.

Homelab Current Form

Sun, 07 Feb 2021 14:35:32 +0200

The first part of the software changes I made was covered in the first part where I explained how and why I started going down this path.

Software choices

This whole journey was about making my home infrastructure better. Part of that was about having a way to more easily create the foundation on which kubernetes would run on, as well as describing my setup as code. Proxmox is a beast in its own right. Features include being able to manage kvm virtual machines and lxc containers, an API that can be used to interact with it programmatically and most importantly being open source. While researching what automation tools could interact with proxmox, I found a community-made terraform provider as well as a set of two ansible community modules (kvm and lxc). After thinking about it a little bit, I wanted to use terraform in a non-cloud environment but also gather some knowledge about it so that’s what I decided to use. I can have my provisioning requirements in a file that terraform understands and store it in git which is exactly what I was going for. Ansible is still in the picture since it is used to set up and configure the debian environment inside the virtual machines. Now that the hypervisor part is covered, let’s move on to running services.

Making it store is harder than making it run

As mentioned in the first part about software changes, I was learning kubernetes and now I could not only wipe and recreate the cluster itself but also the entire virtual machines that it was running on. Some time after starting to use proxmox with terraform I attempted to use rancher to manage kubernetes but ended up ditching it due to various problems with running even basic stuff on it (very likely that it’s a case of PEBCAK, I don’t think rancher is terrible or anything like that).

The problem

However the problem of shared storage continued to taunt me for the following months. There was seemingly nothing that a simple fella could do to use a nice and simple NAS running truenas core as storage that is able to be dynamically provisioned for use by kubernetes. Now, I hear you, “what about nfs-client-provisioner”, someone less familiar with this pile of madness might exclaim. Indeed it does exist and barely work except the helm chart for it is deprecated and it does not work with kubernetes version 1.20 and later since it does not seem to be using CSI drivers.

I really tried

Months of furious and frustrating testing ensued. Not only was I trying to get applications running on kubernetes, I was also fighting with the unexpectedly complex tast of using the storage server I had available.

glusterfs

I’ve gone through basically any and all commonly suggested options for dynamically provisioned storage. Prette early on I tried glusterfs with heketi by making 3 LXC containers and mounting an nfs share to each one that would serve as the brick in the gluster volume. Suffice it to say that it didn’t work and things were getting out of hand.

longhorn

After a bit more research and I found out that longhorn, another project by the authors of k3s, used iscsi to communicate with kubernetes. That could work I thought, except there was no coherent example of how to use it without using targetd for iscsi. Longhorn was starting to look more appealing, I could just put the virtual drives of the VMs on an nfs share, run longhorn inside the VMs to pool all their storage together and call it good enough. No, I could not give up yet. A sub-optimal solution would do if there was no other way but I was convinced something more was out there. My patience and hope were running out but not empty yet.

success

That all changed in early January where during my nearly daily search for possible storage solutions I hadn’t tried, I found out about democratic-csi. This was it. Made to be used with freenas, truenas as well as DIY ZFS setups. The silver bullet was here. Just a few minutes of reading about it and writing my configs, a short(read: long) helm command later and…success! The test pod was using the newly created storageclass that was backed by the nfs share on truenas. After that I rushed to deploy my standard set of gitea, droneci and minecraft to test it out and it was working for real without a hitch.

Conclusion

This was a long journey and it isn’t coming to a close any time soon. The next step is to continuous deployment but that’s an issue for another time. To conclude as briefly as possible, I’ve no put most of the critical pieces of my homelab in a single git repo that I can use to recreate almost all of it from scratch with minimal manual intervention (there are still a few quirks like not having dynamic inventory for ansible and having to manually copy IPs but I can put that aside for now). Despite the imperfections like not having dynamic inventory for ansible or having secrets stored unencrypted in git, I’m very happy with the setup is working. If you missed them, make sure to check out the hardware updates and the first part about software changes for a better of what I’ve been up to.

Homelab Evolved

Sun, 07 Feb 2021 13:53:10 +0200

The hardware side of the changes made to my homelab were covered in a previous post where I also alluded to some software changes.

What came before

This is the first part explaining more in-depth the issues that I had and how I dealt with them. Let’s set the stage first, my homelab was working fine in many ways. It was nice and stable debian install running kvm virtual machines using qemu and libvirt with virt-manager to perform the simpler start/stop tasks. There was really nothing wrong with it, I was exploring different software, running some services on the local network and even hosting a public website on it.

Good enough is not good enough

However as someone interested in infrastructure a few things were bothering me. Sure, I was using docker-compose with traefik for my public website and services. Yes, I was using ansible to manage the configuration of most virtual machines. But “good enough” and “I guess it works” doesn’t cut it. On the upside, I really liked having ZFS snapshots on a 2x2TB mirror. The ability to go back to a point in time where things were not broken had saved me a couple times when I inevitably broke one thing or deleted a file that could be replaced but would take a while. My backups were also on that ZFS mirror, on a different dataset of course, and that also very useful. I knew I wanted ZFS and that there was much room for improvement in regards to handling how I was running services. The single docker-compose virtual machine for the public stuff and another one for local stuff were mostly adequate but the manual management was not. Not to mention that and all my precious stuff was on one box which is well below ideal.

Taking the step

One day I decided it was time to move up in the world. Despite using docker and docker-compose for years I had never dived into kubernetes because it seemed difficult and complex. New syntax, new system with its own architecture and internal structure, new workflow, new everything. I decided to set up 3 VMs and played with default k8s as well as k3s on them. Dipping my toes into it was pretty fun and since nothing depended on it I was free to wipe it and start over. The same issue kept cropping up, having to manually install debian thrice and set up kubernetes just to delete all of it and redo it was again, less than ideal Initially I started writing a playbook before finding out that someone smarter had already gone down this path. Setting up k3s-ansible was easy so there was at least kubernetes setup automation but the whole thing was too thrown together there was no shared storage. However, I was becoming more familiar with kubernetes and learning the concepts as well as how to write yaml manifests.

Conclusion

At some point it became too bothersome, a lot of manual work was involved and basic stuff like copying over ssh keys was also done mostly manually so I had to move on. While all of this was going on, I had started planning what hardware updates I would get which you can read about here: Hardware Updates 2020

Hardware Updates 2020

Fri, 15 Jan 2021 21:10:38 +0200

Getting a real storage server

During 2020 I got a few upgrades to my setup but I don’t remember the exact order of them. The most exciting was a 4U rackmount case that I could use to put up to 6×3.5″ hard drives by default and up to 9 with a 2×5.25″ to 3×3.5″ hotswap bay. Also I bought 3x4TB ironwolf hard drives to put in it initially because my 2x2TB mirror was regularly reaching 95% capacity meaning I had to prune the snapshots regularly and couldn’t use it to store more files on it. This made me warm and fuzzy inside but the old mirror had to sit on the bench for this transition. The power supply in my server had only 4 sata connections which would be enough for a boot SSD and one of the arrays but not everything at once. Initially that wasn’t a problem, installed truenas core on the ssd, set up the 3 4TB hard drives in a raidz1 configuration and started creating datasets and shares.

Adding some extra spice

After a while I wanted to access the data on the old miror which meant having to buy a new power supply so I got one. Namely the Corsair CV550 which has 7 sata power connectors, more than enough to power the five hard drives and the one SSD. Moving on to issue number two, data connections of which the motherboard only had 4 of. Admittedly 50 euros for an extra 4 sata ports seemed excessive but I saved up and acquired a nice and simple pcie card that did the job, no raid controller of course. Alright, we’re cooking with gas now, storage is taken care of.

Dealing with separation of concerns

However, I used to have a multi-purpose server and now it is just a NAS so I can’t run all my services anymore. Fear not because the trusty old and used Dell Optiplex came to the rescue. A shop near my house had one with an i5 2400 and 4gb of ram as well as a 250gb hard drive with windows on it for just 130 euros so I jumped on it. First things first, I can’t be using a mechanical hard drive for the operating system and proxmox doesn’t dual boot so the HDD has to be replaced. Luckily the 240gb SSD from when I originally built my desktop is still alive and it’ll do more than fine for this purpose. The hard drive will be put aside for now but rest assured I do plan to use it at some point.

Serially accessing memories about ram upgrades

More importantly, I got a handle on the ram situation of my NAS. Upgraded that beast from 12 to 16 gigabytes of ram which left the 4gb stick as a spare. Not being one to miss out on some extra ram, the spare got added to the optiplex. Going from 4 to 8 is nice but I want to run more than 2 virtual machines. With that in mind and a deal available, I added an 8gb to the optiplex. So we’re up to 16(8+8) gigabytes for the NAS and 16(2+2+4+8) gigabytes for the server.. An odd configuration but ram is ram and you bet I’ll use it.

Conclusion

Perfect, we had storage and now we have compute as well. How was this hardware used though? Check back later (or read the next post if it’s up) to find out.

Computers Were Faster

Tue, 29 Dec 2020 17:10:41 +0200

I own 3 operational 32-bit computers and I find them fun to play with and useful for software projects. Two of them are socket 475 and one of them is for slot form factor intel cpus. For the slot form factor one, I have a Celeron, a Pentium II and a Pentium III. The socket 478 based ones are much closer to usable for most of my use cases and the other one is fun to tinker with. How can that ““obsolete trash”” be useful, one might rightfully wonder.

My Use Cases

Development use case

Computers with less than 4GB of RAM and 2 cores are completely unusable and nobody should ever touch them, a “muh modern” advocate might cry. However I use those computers at least once a week for various tasks. First off, as a 32-bit docker registry. They don’t need to mix with the rest of the environment since there is no actual dependency between the 32bit and 64bit infrastructure. In addition, I can see how long it takes to build a certain image based on my software, test its performance and experiment with multi-architecture images. I would do the same thing for the raspberry pi but I only have one and that is not enough to warrant the effort. If the above wasn’t enough, they are very useful for testing things such as bare metal provisioning with a mostly DIY setup, how to configure new things I’m interested in such as the aforementioned DIY bare metal provisioning solution as well as new software stacks like ELK or TIG. They are also useful for API load testing and discovering where the bottlenecks are in a piece of software. On a newer multi-core cpu with loads of RAM and fast storage the bottleneck only shows up when the workload origanically overwhelms the software. However on a more limited system it becomes apparent very quickly where the bottleneck could be. For example, if the hard drive access is more than it should be, you hear the damn thing click every time so you know where to look.

Networking Use Case

In addition, old computers make great routers. Routers? They’re old and weak computers you might think. Well, that anemic MIPS or ARM based all-in-one network box that many ISPs provide you with has a CPU that is orders of magnitude slower than either of my 32bit boxes. For the router use case the same thing applies to the amount as well as speed of RAM. In fact, that’s how I originally got into DIY and selfhosting. The case that used to hold my server had one of the 32-bit motherboards and cpu and ram as well as a hard drive. After finding about pfsense and having that computer as a spare, naturally, I gave it a try. A quick install of pfsense later and I was learning about DNS, DHCP, ARP requests, MAC addresses, gateways, how to set up an access point and started my journey into learning about networking. Another use that requires a somewhat low-cost investment is getting a picoPSU or microPSU or something to that extent to power one of those 32-bit machines, connect it to a TV and use it as a dashboard display for monitoring software.

That’s not the end and imagination is the main limit into making this sort of hardware perform a useful task.

Conclusion

Before you go around calling older computers obsolete, check if PEBCAK is the bottleneck. Thank you for reading, if you have a cool project using older hardware, feel free to let me know.

Beyond Docker

Tue, 22 Sep 2020 12:29:52 +0300

Preamble

(Warning: the examples haven’t been fully validated so they’re not guaranteed to work)

On the first part I only talked about docker and functionality included with it by default. This time we are going to look at a few extras. The list includes but is not limited to:

docker-compose since it’s often the next step after standalone docker
a couple tips for Dockerfiles like being extra paranoid about the base image used and how to make sure the thing inside the container is running and listening to requests
setting up a quick and dirty nextcloud + gitea + simple test site and using the traefik to reverse proxy traffic to all of them
having a local docker registry to be less reliant on an always-on internet connection

Sounds exciting, right? Let’s get started then

Prerequisites

MUST :
- Have Linux installed
- Have Docker installed
- Read the first part
OPTIONAL:
- Some basic knowledge of YAML
- PfSense with DNS Forwarder as the DNS server of the network

Docker-compose

We didn’t delve into real multi-container applications last time and that was intentional. The simple site example, reactjs website and golang backend were all self-contained and ran on their own but it’s fairly rare to have something like that running. Many selfhosting-oriented images out there might have an embedded sqlite database suitable for single-user or low use but provide a way to connect to a real database, like mariadb or postgres, when the use exceeds the basic requirements. How would one go about that you might be asking. Write a nice shell script to handle starting/stopping/restarting everything in the right order? That was my first thought too but the answer is docker-compose. Here is where we enter the concept of a set of services, a collection of all images and configuration that a complete application needs to run. This might include a client component, a server component, a database (or two with one being a cache) maybe even multiple server components. All of this can be done in a single YAML file, no long docker run commands, and can be used in quite a few different ways.

Installation

First of all, how do we get that. Following the first-glance upstream documentation and suggestions it might seem like you install it like docker but I suggest a different route. Of course, first priority is through the repository (if applicable). If you can’t, there is a script in the alternative installation options that is placed in /usr/local/bin or wherever else in$PATH we deem appropriate that in turn downloads docker-compose as a container. This is by far my favorite way of getting it and I recommend you give it a try. However, you are welcome to ignore my advice and use the default method for your distro that is mentioned in the installation section of docker’s docs. After it’s installed we can jump into the fun.

First Use

Writing the docker-compose file

For our first multi-container deployment, let’s try gitea since it already has pretty good examples. The first ingredient we need is a file named docker-compose.yaml or docker-compose.yml if we wish it to be automatically recognized by the docker-compose command the same way a file named Dockerfile is recognized by docker build. However, an argument can be give to the -f flag in order to specify a different file if that’s what we wanted to do. For now, we open our favorite editor and start writing.
First up, we need to specify what version we’re using at the very top of the file. This will determine what features we have access to. The latest major version is 3 so let’s go with that:

`1`	`version: "3"`

It needs to be provided as a string which is why it’s surrounded by double quotes. Next up we specify our services, meaning the declaration block for the individual containers and their settings:

1
2
3

version: "3"

  services:

Please note the whitespace indentation, this is how YAML is written. After telling docker-compose that we’re about to define our services, let’s actually define one:

version: "3"

services:

  server:
    image: gitea/gitea
    restart: always

Which means the service is named server which uses the gitea/gitea image and should always be restarted. Below that we can specify some environment variables:

version: "3"

services:

  server:
    image: gitea/gitea
    restart: always
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - DB_TYPE=postgres
      - DB_HOST=db:5432
      - DB_NAME=gitea
      - DB_USER=gitea
      - DB_PASSWD=gitea

The lines below environment that start with a dash are items of a list in YAML, in this case environment variable and their values. The example above has the disadvantage of possibly including passwords and access keys or tokens in this file so you can’t store it in a public git repository. We’ll take a look at how to remedy this later on and we’ll assume it’s fine for now.

Let’s move on to specifying the networks and volumes this container will use (declarations for networks and volumes will be covered later, don’t worry). First a mount point for data and two for sourcing the time from the host system (the :ro means read only so don’t worry about your host files being overwritten or corrupted). This is what the file looks like after those additions:

version: "3"

services:

  server:
    image: gitea/gitea
    restart: always
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - DB_TYPE=postgres
      - DB_HOST=db:5432
      - DB_NAME=gitea
      - DB_USER=gitea
      - DB_PASSWD=gitea
    networks:
      - gitea
    volumes:
      - ./gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

Quite a bit nicer and easier to read than the multiple command-line declarations I think. Now that it’s hooked up to a network, it needs at least one port to expose its web interface on (port 3000) and there is also the ability to ssh into it (using port 22 which is the standard port for it). Here is how ports are mapped in a docker-compose file:

version: "3"

services:

  server:
    image: gitea/gitea
    restart: always
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - DB_TYPE=postgres
      - DB_HOST=db:5432
      - DB_NAME=gitea
      - DB_USER=gitea
      - DB_PASSWD=gitea
    networks:
      - gitea
    volumes:
      - ./gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "222:22"

The above covers the gitea server part but we defined a database to be used so next up we have to add that. The format is the same as above and no new concepts are introduced so here is the file in full as it exists now:

version: "3"

services:

  server:
    image: gitea/gitea
    restart: always
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - DB_TYPE=postgres
      - DB_HOST=db:5432
      - DB_NAME=gitea
      - DB_USER=gitea
      - DB_PASSWD=gitea
    networks:
      - gitea
    volumes:
      - ./gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "222:22"

  db:
    image: postgres:9.6

    restart: always
    environment:
      - POSTGRES_USER=gitea
      - POSTGRES_PASSWORD=gitea
      - POSTGRES_DB=gitea
    networks:
      - gitea
    volumes:
      - ./postgres:/var/lib/postgresql/data

However there is are a few things still missing from the above. First, the gitea network definition. Second, we’re still storing credentials in the docker-compose file in plaintext form. Last, and less apparent, nothing guarantees that the gitea server instance will start only after the database is operational. To achieve this, we will use depends_on. In this case, the server should not start until the database is ready so depends_on will be added to the server declaration:

version: "3"

services:

  server:
    image: gitea/gitea
    restart: always
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - DB_TYPE=postgres
      - DB_HOST=db:5432
      - DB_NAME=gitea
      - DB_USER=gitea
      - DB_PASSWD=gitea
    networks:
      - gitea
    volumes:
      - ./gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "222:22"
    depends_on:
      - db

  db:
    image: postgres:9.6
    restart: always
    environment:
      - POSTGRES_USER=gitea
      - POSTGRES_PASSWORD=gitea
      - POSTGRES_DB=gitea
    networks:
      - gitea
    volumes:
      - ./postgres:/var/lib/postgresql/data

Simple as that, we have now defined a startup dependency and ensured that our containers are started in the correct order. Let’s move on to the network declaration now.

version: "3"

services:

  server:
    image: gitea/gitea
    restart: always
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - DB_TYPE=postgres
      - DB_HOST=db:5432
      - DB_NAME=gitea
      - DB_USER=gitea
      - DB_PASSWD=gitea
    networks:
      - gitea
    volumes:
      - ./gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "222:22"
    depends_on:
      - db

  db:
    image: postgres:9.6
    restart: always
    environment:
      - POSTGRES_USER=gitea
      - POSTGRES_PASSWORD=gitea
      - POSTGRES_DB=gitea
    networks:
      - gitea
    volumes:
      - ./postgres:/var/lib/postgresql/data


networks:
  gitea:
    external: false

So we are defining a network named gitea and specifying that it is not external (meaning it was not created using docker network create gitea but is instead managed by docker-compose). This is all that’s needed to make this run, improvements will be discussed afterwards.

Running the application

After having written our docker-compose.yml or docker-compose.yaml (as mentioned before, both names are acceptable), we have to actually run it. I assume everyone reading installed the docker-compose command-line tool using their preferred method so we can all run docker-compose up or docker-compose up -d to start in detached mode while in the same directory as the docker-compose.yml file. Same as docker, it will pull the images and then bring up the containers. Pointing a web browser to http://localhost:3000 should be enough to see gitea’s front page. We’re up and running!

I’ll have to be a buzzkill and remind you that we’re still storing access credentials inside a file that could and should be in a source code version control repository. Public git repositories are free of charge on the vast majority of code hosting websites and allow other people to reuse our docker-compose YAML files to run services more easily. It becomes clear then that our credentials should be stored outside the docker-compose.yml file. Where could we store them though? Inside .env of course! Docker-compose can read environment variables if there is a file called .env inside the same directory that the docker-compose.yml file is. This means that only that single file can be excluded from version control or encrypted and stored in the repo while keeping everything publicly accessible. Let’s take a look at the .env file then:

UUID=1000
UGID=1000

DBNAME=gitea
DBUSER=gitea
DBPASSWD=gitea

PSTGRSUSR=gitea
PSTGRSPASSWD=gitea
PSTGRSDB=gitea

Simple as that, variable=value pairs is all that’s needed. I avoided underscores and refactored the names a bit to make it clear when the above variables are used. Let’s see how the docker-compose file will look if we want to source the values of the variables we defined:

version: "3"

services:

  server:
    image: gitea/gitea
    restart: always
    environment:
      - USER_UID=$UUID
      - USER_GID=$UGID
      - DB_TYPE=postgres
      - DB_HOST=db:5432
      - DB_NAME=$DBNAME
      - DB_USER=$DBUSER
      - DB_PASSWD=$DBPASSWD
    networks:
      - gitea
    volumes:
      - ./gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "222:22"
    depends_on:
      - db

  db:
    image: postgres:9.6
    restart: always
    environment:
      - POSTGRES_USER=$PSTGRSUSR
      - POSTGRES_PASSWORD=$PSTGRSPASSWD
      - POSTGRES_DB=$PSTGRSDB
    networks:
      - gitea
    volumes:
      - ./postgres:/var/lib/postgresql/data


networks:
  gitea:
    external: false

Pretty easy, right? Just like grabbing the value from variables in the shell. Now all that’s required is to put .env in the repository’s .gitignore so access credentials are sure to be kept safe. Not only that, it’s now possible to use a different .env in production and a different one while developing without changing the docker-compose.yml at all. Feel free to play around with this and see what else you can use this trick for. For now, stop the running containers using docker-compose down if running in detached mode or by hitting Ctrl-C if you started it with docker-compose up and then start it again. We should notice no functional difference other than knowing our credentials are safe.

Personal preference of mine is to set the name (equivalent to --name in docker run) for the containers as well as their hostnames (equivalent to --hostname in docker run), usually to the same value. The final docker-compose.yml would look like this if you wanted to adopt my pet peeve:

version: "3"

services:

  server:
    name: gitea-server
    container_name: gitea-server
    image: gitea/gitea
    restart: always
    environment:
      - USER_UID=$UUID
      - USER_GID=$UGID
      - DB_TYPE=postgres
      - DB_HOST=db:5432
      - DB_NAME=$DBNAME
      - DB_USER=$DBUSER
      - DB_PASSWD=$DBPASSWD
    networks:
      - gitea
    volumes:
      - ./gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "222:22"
    depends_on:
      - db

  db:
    name: gitea-db
    container_name: gitea-db
    image: postgres:9.6
    restart: always
    environment:
      - POSTGRES_USER=$PSTGRSUSR
      - POSTGRES_PASSWORD=$PSTGRSPASSWD
      - POSTGRES_DB=$PSTGRSDB
    networks:
      - gitea
    volumes:
      - ./postgres:/var/lib/postgresql/data
networks:
  gitea:
    external: false

Healthchecks

Docker is unwise as to the actual status of the container, its knowledge by default is limited to the exit code of the command in the ENTRYPOINT or CMD declaration. Fear not though, we can definitely make it a little smarter. Enter the HEALTHCHECK statement. Its syntax is quite easy and it provides better understanding of the program’s status to the container runtime. Admittedly, this is much more useful when working with kubernetes but it never hurts to know if the program is actually working or if it’s hanging but hasn’t fatally exited.

As on the previous part, we’ll start with a simple example. Our old pal nginx will be useful once more. Also this is a great opportunity to learn a new trick. If we start the nginx image in non-detached mode, the web server’s logs will be streamed to our terminal and any commands we type will have no effect since we’re not in a shell. How can we override this and get a shell when we start a new container? The docker run command will be useful here. See, we can redefine the entrypoint and set it to be a shell. Combine that with the -i and -t arguments and we should be off to the races. After a quick docker run --rm -it --entrypoint /bin/bash nginx we get a shell prompt. But hold on, since we redefined the entrypoint, nginx is not actually running (you can check that by installing the procps package and then running top or the old trusty ps aux). Hit Ctrl-C to exit the interactive session (the container will be removed due to --rm) and let’s find a different route.

First, start an nginx container in detached mode docker run --rm -d --name ngx nginx and then with the help of docker exec -it ngx /bin/bash get a shell. Ta-da! Don’t forget, if you don’t give it a nice name you can always use the hash-like name shown in docker container ls, no hindsight required. I walked you through all of this because it’s going to be useful for debugging should you find yourself in a sticky situation and because we’ll demonstrate a common healthcheck command. The one in question is curl --fail http://localhost || exit 1. This will exit with 1 (which is the error/problem code) if there is an error or 0 if there isn’t. Of course we can also replace http://localhost with the URL of our choice. This is a common way to check if a service is still responding normally for obvious reasons, all that is required is curl, a very common utility and we get improved awareness of our program’s status. Let’s see how we can take a Dockerfile from the previous part and add a healthcheck to it. Here is the original (to be used with this project):

FROM node:14.1.0-alpine3.11 as build
WORKDIR /app
COPY . .
RUN npm install --global react-scripts
RUN npm install
RUN npm run build

FROM nginx:1.17.10-alpine
COPY --from=build /app/build /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

And here it is with the healthcheck added:

FROM node:14.1.0-alpine3.11 as build
WORKDIR /app
COPY . .
RUN npm install --global react-scripts
RUN npm install
RUN npm run build

FROM nginx:1.17.10-alpine
COPY --from=build /app/build /usr/share/nginx/html
HEALTHCHECK CMD curl --fail http://localhost || exit 1
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Easy to read and write. It doesn’t end there though. We can tweak a few knobs on it. What if it’s a pretty complex Spring Boot program that takes a couple minutes to start up but after it gets there, we want to check up on it every 15 seconds and allow it 9 seconds and 5 retries to respond since it’s not mission critical? This can all be tweaked. Let’s take those one by one (please note: -f is the same as --fail and Spring Boot runs on port 8080 by default and exposes a health endpoint on /health).

In English:

2 minutes to start up, after that check every 5 seconds

In Dockerfile:

`1`	`HEALTHCHECK --start-period=120s --interval=15s --timeout=9s --retries=5 CMD curl -f http://localhost:8080/health \|\| exit 1`

And that’s it! Now it’s going to run that command and change the container’s health status accordingly. But what if I told you that I don’t recommend using curl for this? Why would that be? If you read the first part, you can see that I used a pretty slim image by picking busybox to base the actual running image. In fact, you can replace that command with no more than 25 lines of Go. What might that look like? I’m glad you asked. Here it is:

package main

import (
    "fmt"
    "net/http"
    "os"
)

func main() {
    URL := os.Getenv("URL")
    if URL == "" {
        URL = "http://localhost"
    }

    fmt.Println("getting:", URL)
    r, err := http.Get(URL)
    if err != nil {
        os.Exit(1)
    }
    fmt.Println("status code:", r.StatusCode)
    if r.StatusCode != 200 {
        os.Exit(1)
    }

    fmt.Println("exit status 0")
    os.Exit(0)
}

And that’s all of it, you can compile the above into a single, tiny, static binary and move into any compatible image. Not to mention that with Go’s amazingly easy cross compilation ability, a quick and easy CGO_ENABLED=0 GOOS=linux GOARCH=arm go build -v gets you a binary to run or a Raspberry Pi for example. Play around with it if you want, I’ve been liking it lately and using it (mostly in images built on the scratch one) instead of using a bigger image that either includes curl or has access to it.

LAN-only setup with reverse proxy

Time to move back to docker-compose. Traefik came to my attention when looking up reverse proxy options. To be precise, this tutorial about traefik v1.x which was the latest version available back then, as well as a new guide for traefik v2.x and I’ve greatly benefited from both. In my view, they’re amazing for a public-facing server (that’s what this site is using) but for LAN-only setups the solutions from the above tutorials seem overcomplicated and overkill. Below I’ll share my docker-compose.yml as well as traefik-related configuration. Feel free to to also look at the raw files in the repository for more help. Specifically, for how to run nextcloud without https which took me a good 20 minutes of searching to find the relevant documentation (always love multiple major version old docs linked from a forum post made at least half a decade ago). However, let’s back up just a bit, one thing at a time.

Reverse Proxies

What are they and what do they do? A proxy is essentially a middleman, in the context of internet traffic, it usually means that it’s taking the traffic you send it and forwarding it somewhere. Flipping this concept, a reverse proxy takes the traffic sent to it and decides where in the servers behind it to send it. So, normal proxy is a middle man for all its clients to reach any server while a reverse proxy is a middle man for all its servers to be reached by a client. Too theoretical? Here is an example then, say my gitea instance is running on my laptop and my blog is running on my desktop. When someone visits git.inherently.xyz they should be redirected to my laptop and if they visit blog.inherently.xyz they should be redirected to my desktop (assuming those things are hosted on the respective devices). It would be possible to open the required ports on both computers and keep opening ports for each new thing I decide to run but it gets out of hand quickly. Here is where a reverse proxy comes in to solve the problem. I can run hundreds of applications or services and only open 1 port, the reverse proxy’s port. The most used reverse proxies are nginx, haproxy and traefik (maybe caddy too). Initially I chose traefik because it was created for a container-focused workload and the excellent tutorial for traefik v1.x that I linked above existed. Now that this is covered, let’s move on to the actual setup.

Writing the docker-compose.yml

First up, make a directory (I recommend $HOME/docker) to put all the relevant files in (this will be known as DOCKDIR for our purposes). Since the basics have been explained, here is where we are going to start from, create a docker-compose.yml inside the DOCKDIR with the following content

version: "3"

services:

  traefik:
    image: traefik:1.7-alpine
    container_name: traefik
    hostname: traefik
    restart: always
    networks:
      - default
      - traefik_proxy
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ${DOCKDIR}/traefik:/etc/traefik


networks:
  traefik_proxy:
    external:
      name: traefik_proxy
  default:
    driver: bridge

You didn’t read this wrong, I’m using the latest release of the v1.x of traefik since the configuration is much simpler. This isn’t fully ready but we’re taking it one step at a time. So, remember what external means? We have to run docker network create traefik_proxy before ever running docker-compose up since it will fail due to the network missing. As for a brief explanation, traefik’s api is running on port 8080 so we expose that and port 80 is where all HTTP traffic will go to. We also bind mount the configuration directory so we can edit traefik’s config easily. Save and exit the file and create a subdirectory named traefik under DOCKDIR to put the traefik.toml in. Said file will include the following:

#options for loglevel are: DEBUG, INFO, WARN, ERROR, FATAL, PANIC
logLevel = "DEBUG"

#turns off traefik's checking of certs
#since we use none, it's fine
InsecureSkipVerify = true

defaultEntryPoints = ["http"]

[entryPoints]
  [entryPoints.http]
  address = ":80"

[api]
  entryPoint = "traefik"
  dashboard = true
  address = ":8080"

[docker]
endpoint = "unix:///var/run/docker.sock"
domain = "docks.localhome"
watch = true
#we're on LAN only and it avoids a label
exposedbydefault = true

First, enabling debug mode to be able to get logs in case something is wrong and turning off certificate checking since it will all be http-only. We set http as the only default entrypoint and then define it addresses on port 80. After that, we enable the api with the dashboard and make it accessible on port 8080. Finally, we add docker-related settings and set all services to be exposed by default without requiring the use of a label for the container. Labels? Let’s go back to the docker-compose.yml and add a couple things. Traefik uses labels for configuring apects of how the reverse proxy will treat an application. For example:

version: "3"

services:

  traefik:
    image: traefik:1.7-alpine
    container_name: traefik
    hostname: traefik
    restart: always
    networks:
      - default
      - traefik_proxy
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ${DOCKDIR}/traefik:/etc/traefik
    labels:
      - "traefik.backend=traefik"
      - "traefik.frontend.rule=Host:traefik.$DOMAINNAME"
      - "traefik.port=8080"
      - "traefik.protocol=http"

networks:
  traefik_proxy:
    external:
      name: traefik_proxy
  default:
    driver: bridge

This sets the name for the backend that handles its traffic to traefik, makes it accessible by using traefik.$DOMAINNAME (we’ll define this in our .env), sets the port that the application runs on as 8080 and defines the protocol as http. This little bit will be added to almost every container with some modifications if it’s a container that should accept traffic from clients and not a database or something else meant for internal use only. I’ll use a container running caddy and one using nginx to demonstrate:

version: "3"

services:

  caddy:
    image: caddy
    restart: always
    networks:
      - traefik_proxy
    labels:
      - "traefik.backend=caddy"
      - "traefik.frontend.rule=Host:web1.$DOMAINNAME"
      - "traefik.port=80"
      - "traefik.protocol=http"

  nginx:
    image: nginx
    restart: always
    networks:
      - traefik_proxy
    labels:
      - "traefik.backend=nginx"
      - "traefik.frontend.rule=Host:web2.$DOMAINNAME"
      - "traefik.port=80"
      - "traefik.protocol=http"

  traefik:
    image: traefik:1.7-alpine
    restart: always
    networks:
      - default
      - traefik_proxy
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ${DOCKDIR}/traefik:/etc/traefik
    labels:
      - "traefik.backend=traefik"
      - "traefik.frontend.rule=Host:traefik.$DOMAINNAME"
      - "traefik.port=8080"
      - "traefik.protocol=http"

networks:
  traefik_proxy:
    external:
      name: traefik_proxy
  default:
    driver: bridge

Fairly simple, right? Only one thing is missing at the moment and that is our .env to define a few commonly used variables. Here it is then:

PUID=1000
PGID=998
TZ="Europe/Athens"
DOCKDIR=/home/user/docker
DOMAINNAME=docks.localhome
MYSQL_ROOT_PASSWORD="Th1s1s_A_verySecurePassword_123"

I defined some commonly used things like a password for databases, timezone, the domain name along with the user and group docker runs as. The last two can be discovered by running id and checking the output. But wait, localhome is not the current local domain and there is no host called docks on this non-existent local domain. Sounds weird but this is not a problem, it will be solved through the DNS server (explained later on in the guide) but for now you can put the IP of the computer and put it in /etc/hosts like below

if it is remote:

1
2
3

192.168.35.18	web1.docks.localhome
192.168.35.18	web2.docks.localhome
192.168.35.18	traefik.docks.localhome

or if you’re running it on the same host you’re accessing it from:

1
2
3

127.0.1.1	web1.docks.localhome
127.0.1.1	web2.docks.localhome
127.0.1.1	traefik.docks.localhome

We need this because traefik responds based on the name that is requested. Time to run docker-compose up -d finally and then visit web1.docks.localhome thenweb2.docks.localhome followed up bytraefik.docks.localhome to check if the setup was successful. Hopefully you see the correct page for each one (web1 is caddy, web2 is nginx and traefik is traefik as expected).

Logs

The above is all fine and dandy but what if things go wrong? A common first action is to look at the logs. Since everything is running inside docker, we could start an interactive terminal and check there but anything that has already been printed will not be shown. Here is where the logs subcommand comes in. While in the same directory as the docker-compose.yml (or by using the -f flag) we can see the logs from all the containers that were launched, including the previous 50 lines, like so:

`1`	`docker-compose -f docker-compose.yml logs --tail=50 -f`

I included the file flag for docker-compose and the follow flag for logs to showcase how they should be positioned since order is important here. This command will print the last 50 lines of logs from all containers that were started from the specified docker-compose.yml and keep following the logs as they’re printed. That’s not what we want to always do because the problematic container might be a single one and the clutter from the rest is distracting. No problem, simply append the specific one that needs to be examined, let’s use traefik as the example:

`1`	`docker-compose -f docker-compose.yml logs --tail=50 -f traefik`

And that’s about it for logs, unlike conventional systems there is not much “unicorn” configuration or at least it should be avoided as much as possible. Keeping the setup easy to replicate is a good thing, the container might restart due to failure and that should be no problem for our operations.

Backups

If there are not at least 2 copies of a file, it might as well not exist. Backups are very important yet often neglected. In this case there is no reason for that to be the case. With the above setup in mind, every piece of data required to replicate the setup is under one directory so all we need is to keep a copy of that on a different hard drive. Using a compressed tarball with preserved permissions or rsync, again preserving the permissions, should be enough for a home setup with a single node. Multi-node setups are not going to be covered here since that would land us in kubernetes territory which is vast enough to warrant its own writeup. What would a simple backup such as the one I suggest above look like then? The following commands are examples that you should tweak depending on your setup and not meant to be copy-pasted and used verbatim:

`1`	`rsync -Pauvr --progress /home/user/docker /mnt/externaldrive/docker-bak-$(date +'%F')`

and

`1`	`tar czvf /mnt/externaldrive/docker-bak-$(date +'%F') /home/user/docker`

Registry

Connecting to dockerhub as well as having an account there can bothersome to impossible in air-gapped setups and having to go to the internet every time you want an image is annoying. Good news, we can run our own registry and cache public images as well as push our own images there. This will require a tiny amount of setup on the clients but nothing that will take more than a few seconds.

Local image storage

First let’s see how we can run a local docker registry using a raw docker command:

`1`	`docker run -d -p 5000:5000 -v ./registry:/var/lib/registry registry:2`

With the above setup, we’ll have a persistent registry running on port 5000 of the host that is running it. For the client setup I’m going to assume everything is on the same system so I will use localhost for the registry address but in reality you’ll be running it on a remote host so feel free to replace localhost with the IP or hostname of the host that the registry is running on in the following examples. On the client’s side create the file /etc/docker/daemon.json (it doesn’t exist by default) and insert the following into it:

{
	"insecure-registries" : [
		"localhost:5000"
	]
}

Don’t worry about the “insecure” part, that’s because it’s not using https. In order to test this out, try pushing an image to it and then pulling it:

docker pull nginx:1.19.2-alpine
docker tag nginx:1.19.2-alpine localhost:5000/ngxalp
docker push localhost:5000/ngxalp
docker image remove nginx:1.19.2-alpine
docker image remove localhost:5000/ngxalp
docker pull localhost:5000/ngxalp

And there we have it, our very own local dockerhub. That’s not where the story ends though. I promised we could also cache images and that’s what we’re going to get into next.

Pull-through cache

This one requires a tiny bit more configuration but still nothing extravagant. Start by editing /etc/docker/registry/config.yml (the registry subdirectory as well as the config.yml file don’t exist by default so you’ll have to create them first) and inserting the following in it:

1
2

proxy:
  remoteurl: https://registry-1.docker.io

and afterwards add an extra option in /etc/docker/daemon.json so the file looks like the one below:

{
	"insecure-registries" : [
		"localhost:5000"
	],
	"registry-mirrors": [
		"http://localhost:5000"
	]
}

Docker

Thu, 04 Jun 2020 09:15:23 +0300

Preamble

Docker is quite popular and has a lot of benefits if used appropriately. This tutorial will hopefully help readers become more familiar with it. You snooze, you lose so let’s get into it.

Prerequisites

MUST:
- Have Linux installed
- Have docker installed
- Internet connection to download images
- Some Linux knowledge
OPTIONAL:
- Dockerhub account to push images to

Concepts and Architecture

Images and Containers

What are images, what are containers and how are they different? Images are a snapshot of a system while containers are a running instance of an image. For example, the nginx docker image has all the necessary bits installed to run nginx. When we start that image, there is now an nginx container running.

To see it action, we can run
docker image ls
to see all docker images on our system. To download a new one, also referred to as pulling an image, we can run docker pull nginx
or if a specific version is required,
docker pull nginx:1.17.10

Images are also immutable meaning they don’t keep any changes you make to them. If we were to go into an ubuntu container, run apt update && apt -y full-upgrade then restart the container, the updates would be lost. To demonstrate it, we can run
docker run --rm -i -t ubuntu
then when we see the shell prompt change, run
touch testfile && ls
followed up by exit. Running
docker run --rm -i -t ubuntu
again followed up by ls shows that the file named testfile does not exist there anymore. The interactive shell won’t really come up again until we get to debugging. Unlike virtual machines, containers are not meant to be used interactively.

Some notes so far:
- The --rm removes the container when the docker command exits (not necessary here but good for testing so we don’t fill up our system with unneeded containers)
- The -i -t gives us an interactive terminal to execute commands in this case (this is not available on every image, for example postgres and nginx show logs
  - In the case of nginx and postgres, even without (-it) they capture the terminal with logging so we use -d to run them in detached mode
  - The --name option gives a (hopefully) human-readable name to the container, especially useful when used with -d
- docker pulls the image we specified (in this case the ubuntu one) if it’s not on the system yet when we use docker run
- There is a special image named scratch if you want to build things from the lowest level allowable

Containers can be started and stopped but it’s more complex than that as is evident from our experimentation and notes this far. We’ll return to using nginx now since most images are going to be centered around the included application. Using
docker run -d --name ngx-alp nginx:alpine
we start an alpine-based nginx container running in detached mode.
After checking the output of
docker container ls
specifically the NAMES column, we can see that it did indeed start and was assigned the desired name. A quick
docker stop ngx-alp
will make it disappear from docker container ls but not from everywhere. If we try to run
docker run -d --name ngx-alp nginx:alpine
then docker will return an error since there is a container with that name but in stopped state. We can confirm that using
docker container ls --all
and then choose to start it again using
docker start ngx-alp
or remove it using
docker rm ngx-alp
Remember, we didn’t use --rm so the container did not get automatically removed after we stopped it. You might have figured it out already but the containers are given random unique names as well as a unique ID, both of which can be used to address them when we want to stop, start or remove them.

Ports

Many applications communicate over ports so how do we deal with that communication and move further along with our nginx example. The port required for a basic “this works” page to load from nginx is 80. However, if you were to run docker run nginx and try to curl http://localhost:80
it would not work. For an analogy with more conventional setups, think of it as port forwarding being required. Port 80 inside the container needs to be mapped to a port on the system in order to become accessible which is controlled by the -p option. I will use distinct ports to make the example more clear so here we go, run docker run -d -p --name ngx 3000:80 nginx
(which will use ``nginx:latest) and then run curl http://localhost:3000 This should return a bit of HTML and internal CSS to indicate that the default nginx page is reachable. We see that port 3000 outside of the container is mapped to port 80 inside of the container. To make sure, we can trydocker run -d -p –name ngx2 4000:80 nginx and thencurl http://localhost:4000 to see that it works and trydocker run -d -p –name ngx2 3000:80 nginx`
to check that it won’t allow us to start it since port 3000 is already used by another container.

Networking

From the get-go, docker has a bridge network that all containers connect to by default but don’t have the perks of name resolution like user-defined bridge networks do. What is a bridge network though and are there other types of networks?
We’ll go through them one by one and explain what they’re useful for.

Bridge

First, bridge networks. By utilizing NAT, in a similar way that a router does, they are able to have containers communicate over them (or not for security reasons). Each container connected to a bridge network gets its own IP address and there are iptables rules added to block or allow traffic between networks and containers (or ports as we saw in the previous section). While the default bridge network that docker creates is mostly fine for testing and messing around, user-defined networks have a few advantages that make them suitable for more proper use such as:
- DNS resolution to make containers able to communicate using names and not IPs which could change - Isolation since a container has to be explicitly added to them - Adding and removing a container can be done on the fly (without starting and stopping) - Configurability such as using a larger MTU and in general tweaking them for the use case - Sharing environment variables - Have effectively all ports exposed between them - Not having to use the deprecated --link flag
So it’s safe to say that using user-defined ones is worth it. How to go about it then? Time to find out.
In order to create a network we just have to run docker network create bridge-net or
docker network create --driver bridge bridge-net
and to delete it we use
docker network rm bridge-net
but how do we use it?
For this, we’ll need two containers. Let’s pick alpine since it includes the ping command. The flag to specify what network to attach the container to is --network so the full commands are as follows

1
2

docker run -dit --name alpine1 --network bridge-net alpine:latest
docker run -dit --name alpine2 --network bridge-net alpine:latest

The above will run two containers in detached mode and enable the interactive terminal. We can connect to that terminal,
docker attach alpine1
after which we get a prompt inside it. Let’s test out that name resolution by pinging it 3 times
ping -c 3 alpine2
which should work wonderfully. This means our containers can talk to eachother using their names while also having randomly assigned addresses from docker’s subnet. In case we forget the network’s details, it is possible to use docker network list
to find the network’s name followed by docker inspect bridge-net and get the network’s details in JSON format.

Macvlan

When integration with the network that the host is connected to is desired, we can use macvlan. Despite VLAN being a part of the name, using them is an option rather than a requirement. Let’s start with the possibly more familiar mode.

Bridge mode

In this mode, the network is bridged to a physical interface on the host and it integrates with the underlying network.

To see it in action, (I’ll assume a fairly standard home network scheme) we can look at

$ ip r
default via 192.168.1.1 dev eth0 metric 2
172.16.0.0/16 dev virbr0 proto kernel scope link src 172.16.0.1 linkdown
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev br-b0ed5d210b62 proto kernel scope link src 172.18.0.1
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.2

to find our interface’s name, eth0 in this case and then create the network docker network create -d macvlan --subnet=192.168.1.0/24 --gateway=192.168.1.1 -o parent=eth0 macvlan-net
Connecting containers to it is done the same way we previously used. For example, docker run -dit --name alpine-on-macvlan --network macvlan-net alpine:latest

802.1q mode

I’m not going to go too in-depth about this mode since the amount of people with level 2 or level 3 managed switches is limited but it deserves at least a brief mention. If you’ve worked with VLANS on linux, you know that new interfaces are created, usually in the format the name of the physical parent followed by a dot . and then the VLAN number. To create a network linked to VLAN 31 on the eth0 interface, we use the same command as before but with a VLAN interface instead like so docker network create -d macvlan --subnet=192.168.1.0/24 --gateway=192.168.1.1 -o parent=eth0.31 macvlan31-net

Storage

So far we have not looked into how we work with storing data.
In the beginning it was stated and demonstrated that any changes made do not persist. How do people use containers to run file servers and databases then? The answer is bind mounts and volumes. Bind mounts map a specified directory on the host to one inside the container. Volumes on the other hand are managed through docker and are a bit more isolated from the host’s functionality. Every time we ran a container, an anonymous volume was created. Instead, we could have created a named volume using docker volume create ngx-vol
docker run --name ngx-with-vol -p 3000:80 --mount source=ngx-vol,target=/usr/share/nginx/html nginx
The volumes are under /var/lib/docker/volumes in case you were wondering. Volumes can be shared between two containers and be mounted read-only but we won’t be going into that right now. With bind mounts, it looks a bit different since we need to pick a directory on the host. If the directory doesn’t exist it will be created for us by docker. Let’s try it using
docker run --name ngx-with-bind-mount -p 3000:80 -v /tmp/ngx:/usr/share/nginx/html nginx
which will mount /tmp/ngx from the host to /usr/share/nginx/html in the container. This gives us an error if we try to curl http://localhost:3000
because the bind mounted host directory doesn’t get filled with the contents of the container’s target directory. There is an alternative syntax for this using the --mount option which is as follows
docker run -d --rm --name ngx-w-alt-bind-mount -p 3000:80 --mount type=bind,source=/tmp/nginx,target=/usr/share/nginx/html nginx
and will not create the directory on the host if it doesn’t exist. That’s the main gist about storage when it comes to docker. The thing most often found in the wild is bind mounts with the -v syntax and named volumes.

Installing

There are multiple ways to install docker-compose. From your distributions repositories, from third-party repositories, from the realease binary, from source, from pip but by far my favorite is as a docker container. Take a look over here if you’re interested, otherwise choose the way you prefer to get it installed to your system. After that, we’re ready to write some YAML.

Usage

Taking advantage of docker-compose is mainly by writing a file that describes how you want your services set up. Just make sure to pay attention to indentation because YAML is very picky about that (be prepared to give the spacebar a proper workout). What images will we work with?

Building Images

All the talk about using docker is good but with kubernetes dominating the industry as far as orchestration, docker finds its home as a way to build images and as a container runtime. How do you build an image then? It all starts with a Dockerfile.

Dockerfile

Similar to a Makefile but used for docker containers instead, the Dockerfile describes how to build your image. The first statement is FROM which specifies which base image to use. Other things you need is at least a little bit of familiarity with the tooling used by the project so you can discern what errors mean. Don’t be intimidated by that though, we’ll be going through it step by step.

Single-stage example build

For our starting example, we’ll use nginx and the following simple index.html:

<!DOCTYPE html>
<html>
	<head>
		<title>First Docker Image</title>
	</head>
	<body>
		<h1>It exists and it works.</h1>
	</body>
</html>

So we create a new file named Dockerfile and put the following in it

`1`	`FROM nginx:1.17.10-alpine`

We could bind-mount our files but shipping them inside the image is we’d need if this was a website we were packing so let’s do it. I’m making the assumption that index.html is in the current working directory

1
2

FROM nginx:1.17.10-alpine
COPY ./index.html /usr/share/nginx/html/

Without a port exposed, it won’t be accessible even if we bind ports to it so let’s do that next

1
2
3

FROM nginx:1.17.10-alpine
COPY ./index.html /usr/share/nginx/html/
EXPOSE 80

The final thing we should put in there is what command will be ran when the container starts. Running as a daemon is not desired since it will be in the foreground so we turn that off.

FROM nginx:1.17.10-alpine
COPY ./index.html /usr/share/nginx/html/
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Now for building the image we can run a simple
docker build -f Dockerfile .
to do the job. The -f specifies what file to use to build the image and . signifies the current working directory as the build context. We don’t necessarily have to specify the file every time, if it is named Dockerfile it will be used automatically.

That’s not fancy enough though. We should probably give it a name and a tag (otherwise it will be tagged with :latest) docker build -t first-site:0.0.1 .
Running the old faithful
docker image ls
should reflect that our custom image was built and tagged appropriately.

Congratulations, you just built your first docker image. But things are never this simple, right? We’re not always going to be working with websites let alone ones that have less than 10 lines of HTML. Let’s try something a bit more realistic. What about…a microservice written in golang exposing an api centered around a list of coffees?

Multi-stage golang example build

I am going to be using a microservice written by following a tutorial for demonstration. Ignore the included dockerfile, our main focus now is to understand the steps required to make images. After cloning the project and entering its directory, delete the included Dockerfile (it’s not as correct as the one we’re going to make anyway) and make an empty one. Just the same as before, we’re going to start with a FROM statement.

`1`	`FROM golang:1.14.2`

Picking version 1.14.2 since that is the latest at the time of writing and the code has been tested to work with it. Next, we’re going to go into the $GOPATH directory, in a subdirectory named src and then in a subdirectory with our project’s name. Note that we use /go/src/microsrv since $GOPATH is resolved to /go in this image.

1
2

FROM golang:1.14.2
WORKDIR /go/src/microsrv

We could have used the RUN statement and done RUN cd $GOPATH/src/microsrv but the WORKDIR command works at the Dockerfile level meaning that all subsequent commands will use it and it also creates the directory in case it doesn’t exist already. After that, it’s time to copy over the project files using COPY

1
2
3

FROM golang:1.14.2
WORKDIR /go/src/microsrv
COPY . .

this copies the contents of our current working directory outside of the container (project’s root) into the current working directory inside the container ($GOPATH/src/microsrv) Following that, we run the command to download all the required dependencies and source code but not build or install the project (that’s due to using -d). Here the RUNstatement is used which executes the command after it inside the container.

FROM golang:1.14.2
WORKDIR /go/src/microsrv
COPY . .
RUN go get -d -v ./...

Of course in the end we want to build and install the project so let’s do that

FROM golang:1.14.2
WORKDIR /go/src/microsrv
COPY . .
RUN go get -d -v ./...
RUN go install -v ./...

Right after that is exposing the port, 9090 in this case

FROM golang:1.14.2
WORKDIR /go/src/microsrv
COPY . .
RUN go get -d -v ./...
RUN go install -v ./...
EXPOSE 9090

The last piece missing is the command to be executed

FROM golang:1.14.2
WORKDIR /go/src/microsrv
COPY . .
RUN go get -d -v ./...
RUN go install -v ./...
EXPOSE 9090
CMD ["microsrv"]

On to building the image now
docker build -t coffee-api:0.0.1 . and then running and testing it docker run --rm -d --name coffee-api -p 9090:9090 coffee-api:0.0.1 curl http://localhost:9090/products
which should return a JSON-formatted object of all the coffees and their public-facing details. This is all fine and dandy but if we check docker image ls we see that the resulting image is about 1GB in size which just won’t fly. We have to optimize but how? One way is to think of what’s needed in compile-time compared to run-time. The entire language is not required after the project has been built and not much if any operating-system level tools are required other than for debugging.

To achive this separation we can use multi-stage builds. The first step is taking out runtime-specific things out of the build stage. What port gets exposed and what command is ran when the container starts is not in the build stage’s scope so let’s remove those.

FROM golang:1.14.2
WORKDIR /go/src/microsrv
COPY . .
RUN go get -d -v ./...
RUN go install -v ./...

Time to build the second stage. First we’ll use an alias for the first stage in order to be able to copy stuff from it.

FROM golang:1.14.2 as build
WORKDIR /go/src/microsrv
COPY . .
RUN go get -d -v ./...
RUN go install -v ./...

With that done, let’s define the second stage inside the same file. We’ll use busybox, a very small image that has just enough to be usable for our case. Since the default image uses glibc as its C library, we’ll specify that tag since busybox doesn’t use it by default.

FROM golang:1.14.2 as build
WORKDIR /go/src/microsrv
COPY . .
RUN go get -d -v ./...
RUN go install -v ./...

FROM busybox:glibc

Here is where the alias comes in handy when copying the resulting binary (located at $GOPATH/bin/microsrv) from the first stage

FROM golang:1.14.2 as build
WORKDIR /go/src/microsrv
COPY . .
RUN go get -d -v ./...
RUN go install -v ./...

FROM busybox:glibc
COPY --from=build /go/bin/microsrv /

And finally we add the two lines that were removed

FROM golang:1.14.2 as build
WORKDIR /go/src/microsrv
COPY . .
RUN go get -d -v ./...
RUN go install -v ./...

FROM busybox:glibc
COPY --from=build /go/bin/microsrv /
EXPOSE 9090
CMD ["/microsrv"]

All that’s left now is to build (I incremented the version number so the image doesn’t steal the previous one’s tagged name but normally you want the docker image version to match the version of the application) docker build -t coffee-api:0.0.2 . then run and test
docker run --rm -d --name coffee-api -p 9090:9090 coffee-api:0.0.1 curl http://localhost:9090/products

If everything went according to plan, a JSON object with the coffees was returned to you and you’ve built your first multi-stage docker image (which you should remember to stop using docker stop coffee-api). We’re not done yet though, maybe it’s time we throw our old pal nginx back into the mix and move from the backend to the frontend.

Multi-stage reactjs example build

Now that our backend is all packed up, time to see how we’d go about doing the same thing with the frontend. This time we have to interact with nodejs during the build but it’s not required for runtime. I think you get an idea of what’s to come so let’s not hesitate.

The project I’ll be using for the following demonstration is a simple one-page counter application. Same as the previous example, we’ll clone it, change into its directory and delete the included Dockerfile since we’ll be remaking it from scratch. As I mentioned, it’s good to know the project’s tooling, in this case we need node and npm for building the application but not for running it. We’ll start with a FROM statement to grab the node image but pre-emptively optimize by going with the alpine flavour. The size comparison between the regular version is as follows and should explain why this is a better choice.

1
2
3

$ docker images | grep node
node                             alpine              0854fcfc1637        2 days ago          117MB
node                             latest              a511eb5c14ec        2 days ago          941MB

Coming in at over 8 times the size of the alpine one, the main image is massive and doesn’t include anything we need and won’t get otherwise. Starting with our Dockerfile then, we pick the latest (at the time of writing) node image based on the latest alpine version and alias it to build since it will be our build stage

`1`	`FROM node:14.1.0-alpine3.11 as build`

We then change our current working directory to /app and move all of our project’s freshly cloned files in there

1
2
3

FROM node:14.1.0-alpine3.11 as build
WORKDIR /app
COPY . .

Now on to getting everything installed and ready for the build. Since it’s a reactjs website and the scripts that will be run (as defined in package.json) are from the react-scripts package (which needs to be installed globally) we will opt for installing that first before the usual npm install. Here is the Dockerfile up until now

FROM node:14.1.0-alpine3.11 as build
WORKDIR /app
COPY . .
RUN npm install --global react-scripts
RUN npm install

To finish up the build stage one last command is required in order to generate the files that nginx will serve

FROM node:14.1.0-alpine3.11 as build
WORKDIR /app
COPY . .
RUN npm install --global react-scripts
RUN npm install
RUN npm run build

There we go then. The resulting files will be inside the container in the /app/build directory. No instructions for exposing any ports or the starting command we specified since that’s for the runtime stage. Speaking of which, you might have already guessed what the rest of the Dockerfile will look like. Start with an nginx alpine image, copy over only the necessary files to it from the build stage, open up port 80 and start nginx in foreground mode. Here is the final result

FROM node:14.1.0-alpine3.11 as build
WORKDIR /app
COPY . .
RUN npm install --global react-scripts
RUN npm install
RUN npm run build

FROM nginx:1.17.10-alpine
COPY --from=build /app/build /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

After we’re done with that, we should actually build the image and give it a descriptive tag
docker build -t reactjs-counters:0.0.1 .
followed up by starting it in order to test docker run --rm -d --name reactjs-counters -p 80:80 reactjs-counters:0.0.1 and then visit
http://localhost
in your browser of choice to see if it works.

Publishing Images

We’ve made our image and sharing is caring so how do we make it available to the world and where are images stored? The go-to place for getting images is dockerhub, a public docker registry. Docker registries are where images are stored and distributed from. You can also run your own if you wish but that’s for another time. Right now, we have two images built and ready to be shipped. Well…almost. We could certainly upload them as they are and then try to pull them but we’d find that we need to specify a tag. That happens because they’re not tagged as :latest which is the default tag. One step at a time though.

DockerHub

First, you should make an account if you don’t have one already by going over on the dockerhub website and signing up. Next up, run docker login in a terminal to, you guessed it, log in. This will allow you to push images to your account. I already got mine so we’re ready to start.

Tagging

Since we just finished making our reactjs-based image, let’s use it here too. Previously we only tagged it as reactjs-counters:0.0.1 but we need something more now. The format of the image name should be username/image:tag. There at least 2 ways to go about this. Either rebuild the image and tag it differently or tag the existing one. I’ll go over both of them.

First, let’s look at the “I wish I’d known sooner” method. As we know, images have a unique identifier. This is under the IMAGE ID column in docker image ls output. A single image can be tagged with different names. So here is the way we’d build it with multiple tags, one of them being the properly versioned one and the other one being latest (don’t forget to substitute username with your actual username):
docker build -t username/reactjs-counters:0.0.1 -t username/reactjs-counters:latest .
And there we have it. Simple enough, ain’t it? But how about tagging an already existing image? Someone might not fancy rebuilding theirs.

That’s what the docker tag command is for. All we have to do is find the image ID from the output of docker image ls and then tag it twice. Here we go then (assuming this is the image ID 7755c1923b8a):

1
2

docker tag 7755c1923b8a username/reactjs-counters:0.0.1
docker tag 7755c1923b8a username/reactjs-counters:latest

Pushing

All that’s left is to docker push the image of our choice and make it available so other people can docker pull it.

1
2

docker push username/reactjs-counters:0.0.1
docker push username/reactjs-counters:latest

And that’s it!
We have successfully taken an application, written a Dockerfile for it, built an image for it and published it to DockerHub!

Outro

Hopefully you’ve learned something useful about docker after reading this, I had fun writing it and looking into things that had faded from my memory. Docker is an interesting technology that can be leveraged in many ways to improve development and production workflows everywhere from a single board computer to a fleet of servers in a datacenter. Let me know what you thought about it and I hope you have a good rest of your day.

Home on inherent site

One Billion Rows Challenge

The challenge

My proposed alternative challenge

Solution in Go

The TempStat struct

Opening the file

Processing the input

Round and Print

Bringing it together in main()

Benchmarks

How to get started

Conclusion

Home Assistant

Getting started

Graphing the worst star’s waste heat

Add-ons

Later expansion

Conclusion

Remote Access Homelab

Tailscale

Cloudflare Tunnel

Conclusion

Talos Linux Setup and Configuration

How the Talos configuration works

Separate secrets

Patches

Conclusion

PXE setup for Talos Linux with Matchbox

Plan

PXE

Talos

Matchbox

Definition of Done

Implementation

How do I do X in Y minutes

Explanation

Conclusion

Homelab 2025 Plans

Intro

Hardware

Software

Kubernetes

Router

Conclusion

State of Homelab 2024

Router

Virtualization

Kubernetes

NAS

Hardware

Conclusion

Virtualized pfSense on ProxMox

Background

Preparation

PfSense VM Setup

Results

New Gentoo

Background

Plan

Installation media

Partitioning and formatting the drive

Stage3

Minor configuration

make.conf

MAKEOPTS

USE

repos.conf

Pseudo-filesystem copy-pasta

Chroot

Tangent: (neo)vi(m)

First encounter with udev

Continuing with the chroot

Masking undesired software

Colonel trouble

Bootloader

Booting into the system

Changing the filesystem

Small configuration and tools

/etc/fstab

The `TempStat` struct

Bringing it together in `main()`

`/etc/fstab`

Basic `main.tf`