Programming on inherent site

One Billion Rows Challenge

Mon, 01 Jun 2026 21:32:29 +0300

A couple of years ago I found an interesting programming challenge that includes reading a lot of data and calculating aggregates on them. It’s called the “One Billion Rows Challenge” or 1brc for short. I decided to take on the challenge to see how far I could get and I’d like to introduce others to it because I had fun participating.

The challenge

As the original challenge blog post and the equivalent github repository README both explain, is the following:

Your mission, should you decide to accept it, is deceptively simple: write a Java program for retrieving temperature measurement values from a text file and calculating the min, mean, and max temperature per weather station. There’s just one caveat: the file has 1,000,000,000 rows!

The text file has a simple structure with one measurement value per row:
1
2
3
4
5
6
Hamburg;12.0
Bulawayo;8.9
Palembang;38.8
St. John's;15.2
Cracow;12.6
...
The program should print out the min, mean, and max values per station, alphabetically ordered like so:
1
{Abha=5.0/18.0/27.4, Abidjan=15.7/26.0/34.1, Abéché=12.1/29.4/35.6, Accra=14.7/26.4/33.1, Addis Ababa=2.1/16.0/24.3, Adelaide=4.1/17.3/29.7, ...}
The goal of the 1BRC challenge is to create the fastest implementation for this task, and while doing so, explore the benefits of modern Java and find out how far you can push this platform. So grab all your (virtual) threads, reach out to the Vector API and SIMD, optimize your GC, leverage AOT compilation, or pull any other trick you can think of.

The rules (I’m quoting the github README here):

Any of these Java distributions may be used:

Any builds provided by SDKMan

Early access builds available on openjdk.net may be used (including EA builds for OpenJDK projects like Valhalla)

Builds on builds.shipilev.net If you want to use a build not available via these channels, reach out to discuss whether it can be considered.

No external library dependencies may be used

Implementations must be provided as a single source file

The computation must happen at application runtime, i.e. you cannot process the measurements file at build time (for instance, when using GraalVM) and just bake the result into the binary

Input value ranges are as follows:

Station name: non null UTF-8 string of min length 1 character and max length 100 bytes, containing neither ; nor \n characters. (i.e. this could be 100 one-byte characters, or 50 two-byte characters, etc.)

Temperature value: non null double between -99.9 (inclusive) and 99.9 (inclusive), always with one fractional digit

There is a maximum of 10,000 unique station names

Line endings in the file are \n characters on all platforms

Implementations must not rely on specifics of a given data set, e.g. any valid station name as per the constraints above and any data distribution (number of measurements per station) must be supported

The rounding of output values must be done using the semantics of IEEE 754 rounding-direction “roundTowardPositive”

This was written from the perspective of the author wanting more people to learn about features in Java versions after 8 and possible performance optimizations that can be made using those features. It also points to alternative compilers and runtimes so people can experiment more with parts of the software stack that might be considered infrastructure and as such neglected by developers. When I read about it, I liked the idea of the challenge. One billion rows is a significant amount, the file itself is 13GB on my system so doing anything with it in a reasonable amount of time is complicated. I mean, even just generating the input is complicated enough, let alone implementing something that reads a 13GB file and computes aggregates in the end. The rules provide details that inform which optimizations we are allowed to make and which ones are off the table.

The premise got me curious enough to sit down and write a program that did the naive thing knowing it was sub-optimal at best. I spent several evenings optimizing it, failing to do so and then trying again. Having seen the leaderboards (both the official ones for Java as well as community-built ones for other languages) I know my solution is not fast and there is room for improvement.

My proposed alternative challenge

I like the rules of the original and I would like to enhance them. Mainly, I’m not interested in a Java-only solution, I’d like everyone to be able to use whatever language they prefer. The restriction of not doing build-time calculations stands and I want to expand it, the measurements file should only be accessed at runtime so something like using Go’s embed.FS is not allowed. Other than that, I would suggest trying to avoid memory mapping the file and using unsafe parts of a language at first and see how far you can get without it. And with that, I think the challenge is explained well enough so we can get started with looking at my take on it.

Solution in Go

My solution for the challenge is located at gitlab.com/insanitywholesale/ongoing/x/1brc. It’s a single file and split into functions with plenty of comments so I don’t forget what I was trying to do which hopefully makes it easier to understand.

The `TempStat` struct

Starting off with the data structure that stores the aggregates for one station, we have the TempStat struct:

// TempStat stores measurement data for a station
type TempStat struct {
	Min   int64
	Max   int64
	Sum   int64
	Count int64
}

This is the final form after optimizations so there are a couple surprising things. First, the numbers are floating point in the input data but here we have integers. This is because it was faster to multiply by 10 (we know that values are in the [-99.9,99.9] range) because comparing integers is faster (but for parsing the opposite is true). Second, this is per-station so we need some place where all the results are stored, this is done in main() in a sync.Map which we will look at afterwards. We also don’t calculate the average directly, we’re storing the sum and count which allows for calculating it once at the end.

Opening the file

To start off, we need to open the file generated from the challenge repository. I haven’t done any optimization here although I’m aware others have memory-mapped the file. This is a simple function that returns a os.File:

// OpenMeasurementsFile reads and returns the file containing measurements
func OpenMeasurementsFile() *os.File {
	// Assume file is next to binary
	fileName := "measurements.txt"
	// Unless it's provided as a CLI argument
	if len(os.Args) > 1 {
		fileName = os.Args[1]
	}
	// After determining the path, open it
	file, err := os.Open(fileName)
	if err != nil {
		// If it doesn't exist just panic, there is no point continuing
		panic(err)
	}
	// And return the handle
	return file
}

It’s fairly obvious this is for a challenge, normally I wouldn’t leave a user-facing stacktrace from panic(). Here it worked well enough to remind me “you don’t have the file here and you didn’t provide it as an argument, plzfix kthx bai!~”.

Processing the input

Now that we’ve opened the file, we have to start reading it and doing our calculations. As a performance optimization, one that most people would do by default, I’m reading the measurement and updating the statistics all once. However, a less obvious one that I implemented was to read more than one line at a time. For this reason I have the ProcessLineGroup function which receives a list of lines as a first argument (through a channel, I’ll explain more in the part about the main() function) and then sends each one of them to the ProcessLine function for the actual work to be done. In this case, ProcessLineGroup() acts like a barrier between the goroutine-aware main() and the unaware ProcessLine(). The wg referenced here is a global variable defined as var wg sync.WaitGroup. In Go, a sync.WaitGroup is a construct used for synchronization. It has a counter which gets incremented by Add(x) and decremented by Done(). When the counter is zero, all goroutines being blocked by the waitgroup will then continue. It also takes a second argument, the results map, to make it accessible to ProcessLine. Here is the code definition for the above:

// ProcessLineGroup reads a batch of lines from the channel and processes them
func ProcessLineGroup(c chan []string, results *sync.Map) {
	// Mark goroutine as being done after ProcessLine exits
	defer wg.Done()

	// Read groups of lines from the channel
	for lines := range c {
		// Process each line in the group
		for _, line := range lines {
			ProcessLine(line, results)
		}
	}
}

Going a bit deeper, we’ll look at ProcessLine(). After a single line, one measurement about one station, has been passed to be processed, let’s see what happens to it. We know the format is station name then a semicolon and then a floating-point temperature reading. The fastest idiomatic way I found for doing this was to find the index of the semicolon and then make two string slices, one starting at the beginning of the line and ending before it and one starting after it and ending at the end of the line. Next, we parse it as a float because it’s faster (I played around with this part quite a bit, it’s been over a year since I wrote the program so I’m not sure how I was parsing it as an int) and also discard any errors to avoid an extra if. Then we get into the conversion that was hinted at by TempStat and store that value in the temporary temperature variable temp Since we have guarantees about the input we can afford to do this without checking, in any other case this would lead to incorrect results. The results.LoadOrStore call either returns the value for the key, station here, or it stores and returns the value we pass as the second argument which is a reference to an instance of TempStat with the min, max, sum set to the only measurement value we have for that station and count set to 1 since we’ve only seen it one time. The return value from the results sync.Map is any so we need to assert its type as a pointer to TempStat. The pointers and references are used so we don’t pass around the concrete instance of the struct which would be more data and therefore slower. At the end of the function there is a simple update of min, max, sum and count for the station. The if statements are separate and look a bit weird, it ended up being a bit faster to do it this way and also less duplicate code than the if..else if..else version. And with all that explained, here is the definition of the ProcessLine function:

// ProcessLine processes a single line and updates temperature stats in the map
func ProcessLine(line string, results *sync.Map) {
	// Locate the index of the delimiter,
	delimiterIndex := strings.Index(line, ";")
	// and split the line according to that.
	station, measurement := line[:delimiterIndex], line[delimiterIndex+1:]
	// To store the measurement, first read it as a float (faster than ParseInt),
	tempFloat, _ := strconv.ParseFloat(measurement, 64)
	// and then since we know it'll be in the [-99.9,99.9] range multiply by 10 and store as int
	temp := int64(tempFloat * 10)

	// Atomic map update using sync.Map
	value, _ := results.LoadOrStore(station, &TempStat{Min: temp, Max: temp, Sum: temp, Count: 1})
	// Get stats for relevant station
	stat := value.(*TempStat)

	// Update station temp stats
	if temp > stat.Max {
		stat.Max = temp
	}
	if temp < stat.Min {
		stat.Min = temp
	}
	stat.Sum += temp
	stat.Count++
}

Round and Print

Grouping the output and the rounding might seem a bit odd but the challenge mentions rounding only with regards to output. There is no other point in the code where rounding is relevant so I’ll explain both of these together. I decided to not invest too much time in this part so it is fairly idiomatic and can most likely be improved.

The Round function just calls out to math.Round which looks like it does the thing the challenge asks for. Being frank, I do not remember why the times 10 and then divide by 10 is in here. There is a special case for zero implemented, I think my result failed validation otherwise.

// Round does roundTowardsPositive
func Round(x float64) float64 {
	rounded := math.Round(x * 10)
	if rounded == -0.0 {
		return 0.0
	}
	return rounded / 10
}

The PrintOutput function is generally pretty boring outside of the sync.Map usage and the int to float conversions. The original input data has floating point measurements, we converted them to integers for processing speed but now we need to calculate the average and provide min/max/avg as floats. We pay the cost of calculating the average once per program run instead of once per line processed which is incredibly benficial when the dataset is one billion measurements. The printing works by getting the station names since that is the unique identifier here. There doesn’t seem to be a better way to do this so a function collects all the station names and stores them in a slice of strings and always returns true to satisfy the type of that argument (func (m *Map) Range(f func(key, value any) bool)). After that, the list of station names is sorted and we construct the output string. For each station, the integer values are converted to floats and reduced by an order of magnitude to normalize them to the original values and then passed through Round(). Last, we print the result to standard output:

// PrintOutput prints the final temperature statistics for each station
func PrintOutput(results *sync.Map) {
	stations := []string{}

	results.Range(func(key, value any) bool {
		stations = append(stations, key.(string))
		return true
	})
	sort.Strings(stations)

	finalResultString := "{"
	for idx, station := range stations {
		stat, _ := results.Load(station)
		tempStats := stat.(*TempStat)

		finalResultString += fmt.Sprintf(
			"%s=%.1f/%.1f/%.1f",
			station,
			Round(float64(tempStats.Min)/10.0),
			Round((float64(tempStats.Sum)/10.0)/float64(tempStats.Count)),
			Round(float64(tempStats.Max)/10.0),
		)

		// Add comma and space everywhere except for the last entry
		if idx != len(stations)-1 {
			finalResultString += ", "
		}
	}
	finalResultString += "}"

	fmt.Println(finalResultString)
}

The if for not adding a comma to the last entry bothers me and could probably be converted to use strings.Replace, probably something like strings.Replace(finalResultString, ", }", "}", 1) but oh well, this is how it is now.

Bringing it together in `main()`

Alright then, we’ve seen the individual puzzle pieces so far and connecting them seems somewhat intuitive but the code in main() was probably the thing I spent most of my time on. This is because there are several parameters that can be tweaked here, especially so because of the use of goroutines (A Tour of Go link). In the big picture, we’re looking to read a file with a billion rows and do a simple calculation as fast as possible. One of the obvious ways to make this fast is to split up the work. Easy to say but hard to do. How do we split the work? In how many parts? How big should those parts be? How many worker processes should we have? Answering all of those took a while, the Go profiler was a huge help here by pointing out where the bottleneck was. I believe I have done a decent job at explaining the code in the comments, take a look:

func main() {
	// Define amount of worker goroutines
	numWorkers := runtime.NumCPU()

	// Set GOMAXPROCS equal to amount of goroutines
	runtime.GOMAXPROCS(numWorkers)

	// Open the measurements file
	file := OpenMeasurementsFile()
	defer file.Close()

	// Create thread-safe map to store station statistics
	results := &sync.Map{}

	// Create a channel to send lines to workers
	lineChan := make(chan []string, 1000)

	// Set number of goroutines to wait for
	wg.Add(numWorkers)
	// Create goroutines based on predefined amount
	for i := 0; i < numWorkers; i++ {
		go ProcessLineGroup(lineChan, results)
	}

	// Set chunk size
	chunkSize := 64 * 1024
	// Create fixed-size buffer over which to send chunks
	buf := make([]byte, chunkSize)
	// Keep any partial line for later processing
	partialLine := ""

	for {
		// Read file into buffer
		n, _ := file.Read(buf)
		if n > 0 {
			// Split the buffer into lines
			chunk := partialLine + string(buf[:n])
			lines := strings.Split(chunk, "\n")

			// Save the last line if it is incomplete
			partialLine = lines[len(lines)-1]

			// Send lines to channel in batches
			if len(lines) > 0 {
				lineChan <- lines[:len(lines)-1]
			}
		} else {
			break
		}
	}

	// Send any remaining partial line
	if len(partialLine) > 0 {
		lineChan <- []string{partialLine}
	}

	// Close the channel after all lines have been processed
	close(lineChan)

	// Wait for all workers to finish
	wg.Wait()

	// Print the final results
	PrintOutput(results)
}

Now that you’ve seen the code, I have some notes in no particular order. The channel mechanism allows main() to send the lines to ProcessLineGroup() runnin in the multiple goroutines which then process the line groups created from a 64 * 1024 chunk of bytes. The goroutines are spawned by the go ProcessLineGroup(lineChan, results) call, prefixing a function call with go runs it in a goroutine. The channel and its buffer size is not pefectly tuned but 1000 seemed to be an okay spot to leave it at. The partial line workaround appears to be unnecessay, I think it’s a leftover.

And that’s my solution! Comparing it with ./calculate_average_baseline.sh > out_expected.txt shows that they are identical.

Benchmarks

I did a quick run using hyperfine, the same tool the original challenge author used. In order to avoid disk caching skewing the results I used the --prepare option to sync the filesystem and drop the disk cache:

cd ~/go/src/ongoing/x/1brc && go build
hyperfine --warmup 0 --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' './1brc ~/gitty/1brc/measurements.txt'

Time (mean ± σ):     42.469 s ±  0.654 s    [User: 118.403 s, System: 8.892 s]
Range (min … max):   41.712 s … 43.710 s    10 runs

So I’m at about 42-43 seconds. Not amazing but it’s at least under 1 minute.

How to get started

Looking at READMEs and figuring out scripts takes a couple of minutes so I think a short how to might help lower the barrier to entry. So here are the setup steps:

git clone https://github.com/gunnarmorling/1brc
./mvnw clean verify
./create_measurements.sh 1000000000
./calculate_average_baseline.sh > out_expected.txt

Now you have the measurements file and the expected output to compare to. You can save your solution’s output to a file (e.g. my_output.txt) and use something like diff --suppress-common-lines out_expected.txt my_output.txt to see if you got it right. After you’ve checked for correctness, it’s time to optimize and then re-check that you’re producing correct results but faster! You can use the hypefine command I used as well to check for speed.

Conclusion

My shell history says I started working on this challenge October 2024. I don’t know why almost 2 years later it dawned on me to write a blog post about it but I hope you decide to take it on and have as much fun as I did.

How This Website Is Made

Sat, 20 Nov 2021 20:56:19 +0200

Intro

A significant part of having my own site is the way it’s made behind the scenes. This has changed in a couple different ways for the duration of its existence and in this post I’ll try to explain how and why.

Beginning

When I first bought inherently.xyz to host my own website I wanted my site to be built in the way that I’d like other sites to be built. The main requirements were:

a theme that was mine
basically no javascript
nothing loading from third-party places
code that is easy to understand
readable in the devtools
reasonably small and fast
able to be read on a lot of devices and browsers
over the top way to host it

I don’t think I have a repository where I developed it but this is an archive of some version of it. I’ll retell the story of its existence so you don’t have to look at the source but it’s there if you want to take a look.

Evaluation

In my humble opinion I did fairly well to cover the requirements I had but not as well as I could have. However not all was great. For one, I had used bootstrap, the CSS framework, to make the site. Because I used the navbar hamburger/button/dropdown thing it required some javascript/jquery which was unfortunate. It was my goal to get rid of this but from reading a github issue about it the team behind bootstrap did not feel like providing that functionality through a CSS trick was something they were willing to do.

The theme was mostly mine outside of boostrap and it was adapted from someone else’s CSS colorscheme. As for the code itself, every page consisted of hand-written HTML so that part was easy to read but the minified bootstrap files, aside from being pretty damn big, were unreadable. At least the bootstrap files were hosted on my own server and no third-party anything loading so I can check that off the list. Additionally, to make the site I had created a file called base.html containing the basic template of a page which I would manually copy to create a new page and then rsync the files from my computer to the server. Surprisingly enough, this worked pretty well although changing code that touched multiple pages like the navigation was somewhat cumbersome. Reading the website through a lesser known web browser like netsurf, surf or any text-mode web browser was possible so that requirement was basically 100% satisfied.

Since the beginning of this site I’ve been using docker-compose to host it. For this iteration I was using a simple nginx image and mapping the directory on the filesystem to /usr/share/nginx/html inside the container. Traefik was running in front of it acting as a reverse proxy and to support https. Overkill for a site that has handwritten HTML? Check.

Improvements

There was quite a bit of way to go as you can gather from the above. The main pain points were the existence of javascript causing the navigation dropdown to not work on mobile when javascript was disabled, the performance metrics related to using bootstrap in general, dependency on bootstrap and lack of major customization from me. The raw HTML workflow was mildly annoying but it never stopped me from writing anything.

Solutions

I spent quite a long time trying to find a workaround for the nav dropdown and eventually stumbled into a solution in a post I can’t find anymore, I’ll update this if I do. That worked fine enough and allowed me to get rid of the jquery dependency that, funnily enough, had a vulnerability announced about it while I was rewriting my nav code. Bootstrap was also slowing down everything according to all benchmarks I ran.

This was all fairly fine for quite a while. I kept working on the site, writing stuff and having my own little space on the internet. The curse of being into tech and knowing how to make a website even as basic as this was that I wanted to rewrite it. That’s when I started exploring and found out about hugo, a fast static site generator (SSG for short) written in Go. Given that I was interested in Go at the time of this rewrite and that I knew how the templates worked more or less I decided to give it a try.

Hugo

My first idea was to just port it to hugo and then improve it and quickly realized what a mistake that would be. Why do one thing you’ve never done before if you can do two, right? Hugo is a static site generator meaning that it creates the files required for the site once and that’s it, the site is not dynamically put together on the client. As the author, you mostly write markdown files that are turned into html at build time and then you can put those html files anywhere you’d like. This is what the site uses to this day and you find the source code for it here.

Transitioning

The real work started when I decided I’d make my own theme too. I didn’t just have to move my content and page structure to it but write some nice fresh CSS too since I decided I’ve had enough of bootstrap.

Directory Structure

The first steps was examining the directory structure and finding the most low level template file which is layouts/_default/baseof.html and making my way up. Inside there I defined the very basics of what the HTML would look like, imported my CSS file which at this point had about 5 lines in it and moved on to the other fundamental pages. These are layouts/_default/single.html and layouts/_default/list.html, the former being about a “one of it’s kind” page and the latter being an aggregation. The example here is a blog post and the list of all the blog posts. They are based on baseof.html and in my case are very short, consisting only of the barebones required to implement these types of pages. This is because they are overwritten by category-specific single.html and list.html which is how I make blog posts have a table of contents and how I make the blog section look different than the tutorials section.

Layouts

Speaking of sections, the first thing I got up and running was the blog since that was the easiest one with lots of documentation and examples. The second priority was the front page which turned out to be a little bit more complicated. You put the content for it inside content/_index.md and the layout-related stuff inside layouts/index.html. In hindsight it’s fairly intuitive but it took a few minutes of digging when I was less familiar. For each sub-category you can basically include no layout files in which case the list.html and single.html from layouts/_default will be used or you can write ones that will override them. As an example, in layouts/tutorials I have both a list.html and a single.html to build the tutorial-related pages in their own way.

Layouts are pretty cool and probably my favorite thing is partial layouts. These allow you to put a snippet in a file, let’s say footer.html and then include it using {{ partial "footer" . }} in another template. One other example is the contact form, I just made a layouts/contact/list.html that has the line {{ partial "comments.html" . }} and this way I can keep the template cleaner and allow reusing that comment submission form elsewhere too. So I just went around creating my categories, writing specific list or single templates for them if required and throwing some pieces of reusable code in partial layout templates.

CSS

The most involved part was by far the CSS. I have to say that if it’s not obvious by now I’m not a “frontend person” per se, I just want something that looks clean, fast and readable. As a disclaimer, I’m not a fan of the size of the fonts which are by far the biggest element right now but I’m a huge fan of the fira font family (with the exception of fira code cause I hate ligatures) but they work well enough. It took the majority of the time and had the most updates, tweaks and fixes by far compared to anything else. Essentially nothing from the original CSS is the same except the colors which was a long process but a necessary one to get where I wanted to be. In the end I’m satisfied with it, I think it looks fairly unique and serves its purpose well while coming in at under 200 lines (it is way less in reality but the number balloons up because of the way I format the code).

Docker

This time I went with a custom docker image so I could have more control and no filesystem bind mounts. The main problem is that the hugo team does not publish an official docker image so I had to use the following to get hugo installed in the container in order to generate the site (most current Dockerfile is in the git repository, check there first before copying):

FROM golang:1.16 as buildsite
ENV CGO_ENABLED 0
WORKDIR /go/src
RUN git clone https://github.com/gohugoio/hugo.git
WORKDIR /go/src/hugo
RUN go install -v
WORKDIR /go/src/inheresite-hugo
COPY . .
RUN hugo

FROM nginx:alpine
COPY default.conf /etc/nginx/conf.d/
COPY --from=buildsite /go/src/inheresite-hugo/public /usr/share/nginx/html

This means I compile hugo every time I want to upload a post but it goes fairly fast and gives a bit of time to reflect on what I’ve written. If this is not for you, use either an ubuntu image and install it with apt if you don’t need the latest features or an arch one if you need the latest release for some reason. Using third-party images is a last resort but those are technically also usable for this purpose however I’d advise against using images from random people.

Impressions

After a long while with hugo I think I’ll be sticking with it. It allows for some pretty clean code, I’m partial to Go templating and it has served me well. I’d like to look into some small improvements such as merging tutorials into blog and just displaying them in a different way but that’s a topic for another day. The full server-side syntax highlighting is awesome too, I thought I’d need to cave and use some javascript thing to do it but thankfully it worked out great and I found a code theme that matches the rest of the site. If you want to get started with it, just make sure you know some basic HTML and have some patience, the rest can be figured out in an evening.

Conclusion

The site you’re currently looking at has been a while in the making and there are still a couple items on my TODO.md for it but I’m happy enough with it for now. You might want to check out the comment microservice I wrote for use with the comment form mentioned a bit above, it’s far from perfect but it gets the job done. Thank you for reading this, I hope you enjoyed it and maybe even learned something.

Go API Part 3

Sat, 10 Jul 2021 15:56:21 +0300

Previous relevant tutorial

The development of the CRUD REST API was described in part 2 and the introduction to the series can be found in part 1 so read those first if you’re new.

Intro

After creating a minimally functional API, we’re going to add persistent storage to it in the form of a database. In this case it is going to be postgres running inside docker for ease of installation and use.

Starting point

Here is where we left off last time:

// main.go
package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
	"strconv"
)

type DeliveryDriver struct {
	FirstName string `json:"firstname"`
	LastName  string `json:"lastname"`
}

type Delivery struct {
	OrderNumber      int            `json:"ordernumber"`
	City             string         `json:"city"`
	Zipcode          string         `json:"zipcode"`
	Address          string         `json:"address"`
	Phone1           string         `json:"phone1"`
	Phone2           string         `json:"phone2,omitempty"`
	Cancelled        bool           `json:"cancelled"`
	Delivered        bool           `json:"delivered"`
	DeliveryAttempts int            `json:"deliveryattempts"`
	Driver           DeliveryDriver `json:"deliverydriver"`
}

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{
	&Delivery{
		OrderNumber:      1,
		City:             "Here",
		Zipcode:          "52011",
		Address:          "Home",
		Phone1:           "6945123789",
		Phone2:           "2313722903",
		Cancelled: false,
		Delivered:        false,
		DeliveryAttempts: 0,
		Driver: DeliveryDriver{
			FirstName: "Mhtsos",
			LastName:  "Iwannou",
		},
	},
	&Delivery{
		OrderNumber:      2,
		City:             "There",
		Zipcode:          "1701",
		Address:          "Office",
		Phone1:           "6932728091",
		Cancelled: false,
		Delivered:        true,
		DeliveryAttempts: 1,
		Driver: DeliveryDriver{
			FirstName: "Lucas",
			LastName:  "Johnson",
		},
	},
	&Delivery{
		OrderNumber:      3,
		City:             "FarAway",
		Zipcode:          "920639",
		Address:          "Island",
		Phone1:           "6900777123",
		Cancelled: true,
		Delivered:        false,
		DeliveryAttempts: 24,
		Driver: DeliveryDriver{
			FirstName: "Pilotos",
			LastName:  "Aeroplanou",
		},
	},
}

// Constant for Bad Request
const BadReq string = `{"error": "bad request"}`
// Constant for Not Found
const NotFound string = `{"error": "not found"}`

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(deliveries)
}

// Read a specific delivery
func GetDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

// Create a new delivery
func AddDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	d := &Delivery{}
	// because deliveries start from 1
	orderNum := int(len(deliveries)) + 1
	err := json.NewDecoder(r.Body).Decode(d)
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	// Override whatever the sender had sent
	d.OrderNumber = orderNum
	// Append the delivery to deliveries list
	deliveries = append(deliveries, d)
	w.WriteHeader(http.StatusCreated)
	// Send back the delivery that was saved
	json.NewEncoder(w).Encode(d)
	return
}

// Update an existing delivery
func UpdateDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			err := json.NewDecoder(r.Body).Decode(d)
			if err != nil {
				w.WriteHeader(http.StatusBadRequest)
				w.Write([]byte(BadReq))
				return
			}
			d.OrderNumber = orderNum
			w.WriteHeader(http.StatusAccepted)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

// Delete a delivery (not really)
func DeleteDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			d.Cancelled = true
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

func main() {
	// Set up router
	router := mux.NewRouter()
	// Set up subrouter for api version 1
	apiV1 := router.PathPrefix("/api/v1").Subrouter()
	// Set up routes
	apiV1.HandleFunc("/deliveries", GetAllDeliveries).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery/{ordernumber}", GetDelivery).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery", AddDelivery).Methods(http.MethodPost)
	apiV1.HandleFunc("/delivery/{ordernumber}", UpdateDelivery).Methods(http.MethodPut)
	apiV1.HandleFunc("/delivery/{ordernumber}", DeleteDelivery).Methods(http.MethodDelete)
	// Start http server
	log.Fatal(http.ListenAndServe(":8000", router))
}

Restructuring

In my opinion a good first move is to abstract away the specifics of the database before moving further. Some might see it as a premature optimization but I believe it’s important to not mix the concerns of storing data with the API. We will do that by having an interface that many different data storage backends can satisfy. Initially we will remove the specifics of the simple list we used and make it more generalized. Up until now all the code was in a single file and it’s kind of crowded with data models, API routes, API functions and data storage so that will change.

Important note

Since we will be splitting the contents of the main package over different files we won’t be able to run our program using go run main.go. All the .go files need to be included so the command should instead be go run *.go

Models

First off let’s cut and then paste the models in their own file called models.go next to main.go like so:

// models.go
package main

type DeliveryDriver struct {
	FirstName string `json:"firstname"`
	LastName  string `json:"lastname"`
}

type Delivery struct {
	OrderNumber      int            `json:"ordernumber"`
	City             string         `json:"city"`
	Zipcode          string         `json:"zipcode"`
	Address          string         `json:"address"`
	Phone1           string         `json:"phone1"`
	Phone2           string         `json:"phone2,omitempty"`
	Cancelled        bool           `json:"cancelled"`
	Delivered        bool           `json:"delivered"`
	DeliveryAttempts int            `json:"deliveryattempts"`
	Driver           DeliveryDriver `json:"deliverydriver"`
}

Simple enough I’d say. Remember to delete the code from main.go otherwise the compiler will error out. Next let’s handle the API stuff.

API

In a new file called api.go we will first copy the relevant functions and then do a little refactoring like renaming main() since we can’t have two of them as well as making that function return something that can be easily used by the actual main() function. In this case we’ll just return the router and let the code in main.go handle what port it will run at. Once again, remember to delete this code from main.go otherwise it won’t run.

// api.go
package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"net/http"
	"strconv"
)

// Constant for Bad Request
const BadReq string = `{"error": "bad request"}`
// Constant for Not Found
const NotFound string = `{"error": "not found"}`

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(deliveries)
}

// Read a specific delivery
func GetDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

// Create a new delivery
func AddDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	d := &Delivery{}
	// because deliveries start from 1
	orderNum := int(len(deliveries)) + 1
	err := json.NewDecoder(r.Body).Decode(d)
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	// Override whatever the sender had sent
	d.OrderNumber = orderNum
	// Append the delivery to deliveries list
	deliveries = append(deliveries, d)
	w.WriteHeader(http.StatusCreated)
	// Send back the delivery that was saved
	json.NewEncoder(w).Encode(d)
	return
}

// Update an existing delivery
func UpdateDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			err := json.NewDecoder(r.Body).Decode(d)
			if err != nil {
				w.WriteHeader(http.StatusBadRequest)
				w.Write([]byte(BadReq))
				return
			}
			d.OrderNumber = orderNum
			w.WriteHeader(http.StatusAccepted)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

// Delete a delivery (not really)
func DeleteDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			d.Cancelled = true
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

func MakeRouter() http.Handler {
	// Set up router
	router := mux.NewRouter()
	// Set up subrouter for api version 1
	apiV1 := router.PathPrefix("/api/v1").Subrouter()
	// Set up routes
	apiV1.HandleFunc("/deliveries", GetAllDeliveries).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery/{ordernumber}", GetDelivery).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery", AddDelivery).Methods(http.MethodPost)
	apiV1.HandleFunc("/delivery/{ordernumber}", UpdateDelivery).Methods(http.MethodPut)
	apiV1.HandleFunc("/delivery/{ordernumber}", DeleteDelivery).Methods(http.MethodDelete)
	return router
}

Looking nice and neat. We should probably tinker with main.go too though.

Main after API restructure

Let’s adjust main.go to fit with the changes we’ve made. The list stuff is still here (for now) but the line count is greatly reduced:

// main.go
package main

import (
	"log"
	"net/http"
)

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{
	&Delivery{
		OrderNumber:      1,
		City:             "Here",
		Zipcode:          "52011",
		Address:          "Home",
		Phone1:           "6945123789",
		Phone2:           "2313722903",
		Cancelled: false,
		Delivered:        false,
		DeliveryAttempts: 0,
		Driver: DeliveryDriver{
			FirstName: "Mhtsos",
			LastName:  "Iwannou",
		},
	},
	&Delivery{
		OrderNumber:      2,
		City:             "There",
		Zipcode:          "1701",
		Address:          "Office",
		Phone1:           "6932728091",
		Cancelled: false,
		Delivered:        true,
		DeliveryAttempts: 1,
		Driver: DeliveryDriver{
			FirstName: "Lucas",
			LastName:  "Johnson",
		},
	},
	&Delivery{
		OrderNumber:      3,
		City:             "FarAway",
		Zipcode:          "920639",
		Address:          "Island",
		Phone1:           "6900777123",
		Cancelled: true,
		Delivered:        false,
		DeliveryAttempts: 24,
		Driver: DeliveryDriver{
			FirstName: "Pilotos",
			LastName:  "Aeroplanou",
		},
	},
}

func main() {
	router := MakeRouter()
	log.Fatal(http.ListenAndServe(":8000", router))
}

Since we’re cleaning things up and are planning to add a database, time to remove the list and put it in its own file.

List Restructure

Our main.go is getting emptier so let’s remove the list stuff and put it in its own file called listdb.go

// listdb.go
package main

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{
	&Delivery{
		OrderNumber:      1,
		City:             "Here",
		Zipcode:          "52011",
		Address:          "Home",
		Phone1:           "6945123789",
		Phone2:           "2313722903",
		Cancelled: false,
		Delivered:        false,
		DeliveryAttempts: 0,
		Driver: DeliveryDriver{
			FirstName: "Mhtsos",
			LastName:  "Iwannou",
		},
	},
	&Delivery{
		OrderNumber:      2,
		City:             "There",
		Zipcode:          "1701",
		Address:          "Office",
		Phone1:           "6932728091",
		Cancelled: false,
		Delivered:        true,
		DeliveryAttempts: 1,
		Driver: DeliveryDriver{
			FirstName: "Lucas",
			LastName:  "Johnson",
		},
	},
	&Delivery{
		OrderNumber:      3,
		City:             "FarAway",
		Zipcode:          "920639",
		Address:          "Island",
		Phone1:           "6900777123",
		Cancelled: true,
		Delivered:        false,
		DeliveryAttempts: 24,
		Driver: DeliveryDriver{
			FirstName: "Pilotos",
			LastName:  "Aeroplanou",
		},
	},
}

Nice and easy, we radically reduced the code inside main.go and separated our code in nicely named files.

Main after list database restructure

Take a peek at main.go after all the changes. You will notice that it is a lot shorter and only deals with running our API server and not much else:

package main

import (
	"log"
	"net/http"
)

func main() {
	router := MakeRouter()
	log.Fatal(http.ListenAndServe(":8000", router))
}

Database

This is a big addition so we’ll have to think about it a little bit. In my opinion a good first move is to abstract away the specifics of the database. Some might see it as a premature optimization but I believe it’s important to not mix the concerns of storing data with the API. We will do that by having an interface that many different data storage backends can satisfy.

Interface

The fake database we used in part 2 is mostly in its own file called listdb.go but there are still implementation-specific details of it inside api.go. Before ripping out what exists, we’ll define our interface. Due to what it is, it fits best inside models.go. This is a simple interface of something that can essentially perform the same actions as the API except it only deals with talking to the database. Take a look at what is in models.go now:

// models.go
package main

type DeliveryDB interface {
	ReturnAll() ([]*Delivery, error)
	ReturnOne(orderNumber int) (*Delivery, error)
	Store(*Delivery) (*Delivery, error)
	Change(orderNumber int, del *Delivery) (*Delivery, error)
	Remove(orderNumber int) error
}

type DeliveryDriver struct {
	FirstName string `json:"firstname"`
	LastName  string `json:"lastname"`
}

type Delivery struct {
	OrderNumber      int            `json:"ordernumber"`
	City             string         `json:"city"`
	Zipcode          string         `json:"zipcode"`
	Address          string         `json:"address"`
	Phone1           string         `json:"phone1"`
	Phone2           string         `json:"phone2,omitempty"`
	Cancelled        bool           `json:"cancelled"`
	Delivered        bool           `json:"delivered"`
	DeliveryAttempts int            `json:"deliveryattempts"`
	Driver           DeliveryDriver `json:"deliverydriver"`
}

Short and sweet, does what it should. How do we use this abstraction though? For this we’ll need to move quite a few things out of api.go into listdb.go.

Separating concerns

We will start by moving the things related to data storage out of api.go into listdb.go. This will be done so we can implement the DeliveryDB interface. Before the actual work, take a look at a shortened version of the data-related parts of api.go:

// api.go

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	json.NewEncoder(w).Encode(deliveries)
}

// Read a specific delivery
func GetDelivery(w http.ResponseWriter, r *http.Request) {
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
}

// Create a new delivery
func AddDelivery(w http.ResponseWriter, r *http.Request) {
	// Override whatever the sender had sent
	d.OrderNumber = orderNum
	// Append the delivery to deliveries list
	deliveries = append(deliveries, d)
}

// Update an existing delivery
func UpdateDelivery(w http.ResponseWriter, r *http.Request) {
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			err := json.NewDecoder(r.Body).Decode(d)
			if err != nil {
				w.WriteHeader(http.StatusBadRequest)
				w.Write([]byte(BadReq))
				return
			}
			d.OrderNumber = orderNum
			w.WriteHeader(http.StatusAccepted)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
}

// Delete a delivery (not really)
func DeleteDelivery(w http.ResponseWriter, r *http.Request) {
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			d.Cancelled = true
			json.NewEncoder(w).Encode(d)
			return
		}
	}
}

These is what we will grab and reshape to be the implementation of the DeliveryDB interface inside listdb.go. The ReturnAll, ReturnOne, Store, Change and Remove functions should be implemented on a type, in this case, deliveryList.

List database

There isn’t much more to say so here it is:

// listdb.go
package main

import (
	"errors"
)

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{
	&Delivery{
		OrderNumber:      1,
		City:             "Here",
		Zipcode:          "52011",
		Address:          "Home",
		Phone1:           "6945123789",
		Phone2:           "2313722903",
		Cancelled:        false,
		Delivered:        false,
		DeliveryAttempts: 0,
		Driver: DeliveryDriver{
			FirstName: "Mhtsos",
			LastName:  "Iwannou",
		},
	},
	&Delivery{
		OrderNumber:      2,
		City:             "There",
		Zipcode:          "1701",
		Address:          "Office",
		Phone1:           "6932728091",
		Cancelled:        false,
		Delivered:        true,
		DeliveryAttempts: 1,
		Driver: DeliveryDriver{
			FirstName: "Lucas",
			LastName:  "Johnson",
		},
	},
	&Delivery{
		OrderNumber:      3,
		City:             "FarAway",
		Zipcode:          "920639",
		Address:          "Island",
		Phone1:           "6900777123",
		Cancelled:        true,
		Delivered:        false,
		DeliveryAttempts: 24,
		Driver: DeliveryDriver{
			FirstName: "Pilotos",
			LastName:  "Aeroplanou",
		},
	},
}

func NewListDatabase() deliveryList {
	return deliveries
}

func (dl deliveryList) ReturnAll() ([]*Delivery, error) {
	return deliveries, nil
}

func (dl deliveryList) ReturnOne(orderNumber int) (*Delivery, error) {
	for _, d := range deliveries {
		if d.OrderNumber == orderNumber {
			return d, nil
		}
	}
	return nil, errors.New(NotFound)
}

func (dl deliveryList) Store(d *Delivery) (*Delivery, error) {
	d.OrderNumber = int(len(deliveries))
	deliveries = append(deliveries, d)
	return d, nil
}

func (dl deliveryList) Change(orderNumber int, del *Delivery) (*Delivery, error) {
	for _, d := range deliveries {
		if d.OrderNumber == orderNumber {
			del.OrderNumber = orderNumber
			d = del
			return d, nil
		}
	}
	return nil, errors.New(NotFound)
}

func (dl deliveryList) Remove(orderNumber int) error {
	for _, d := range deliveries {
		if d.OrderNumber == orderNumber {
			d.Cancelled = true
			return nil
		}
	}
	return errors.New(NotFound)
}

We’ve moved everything related to data-handling out of the API code and have a mock database to work with. Obviously it’s only in memory so if we stop and restart the application any changes made to the data isn’t saved anywhere.

Refactor API code

Before moving on to implementing a different database let’s change the API code to use the generalized interface. I also changed a couple stuff related to HTTP error codes because some were incorrect and some because most clients expect a 200 response if a request is successful. At any rate, let’s look at the refactored api.go:

package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"net/http"
	"strconv"
)

// Constant for Bad Request
const BadReq string = `{"error": "bad request"}`

// Constant for Not Found
const NotFound string = `{"error": "not found"}`

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	dels, err := deliverydb.ReturnAll()
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(dels)
	return
}

// Read a specific delivery
func GetDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	del, err := deliverydb.ReturnOne(orderNum)
	if err != nil {
		w.WriteHeader(http.StatusNotFound)
		w.Write([]byte(NotFound))
		return
	}
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(del)
	return
}

// Create a new delivery
func AddDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	d := &Delivery{}
	err := json.NewDecoder(r.Body).Decode(d)
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	del, err := deliverydb.Store(d)
	if err != nil {
		w.WriteHeader(http.StatusNotFound)
		w.Write([]byte(NotFound))
		return
	}
	w.WriteHeader(http.StatusCreated)
	json.NewEncoder(w).Encode(del)
	return
}

// Update an existing delivery
func UpdateDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	d := &Delivery{}
	err = json.NewDecoder(r.Body).Decode(d)
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	del, err := deliverydb.Change(orderNum, d)
	if err != nil {
		w.WriteHeader(http.StatusNotFound)
		w.Write([]byte(NotFound))
		return
	}
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(del)
	return
}

// Delete a delivery (not really)
func DeleteDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	err = deliverydb.Remove(orderNum)
	if err != nil {
		w.WriteHeader(http.StatusNotFound)
		w.Write([]byte(NotFound))
		return
	}
	w.WriteHeader(http.StatusOK)
	return
}

func MakeRouter() http.Handler {
	// Set up router
	router := mux.NewRouter()
	// Set up subrouter for api version 1
	apiV1 := router.PathPrefix("/api/v1").Subrouter()
	// Set up routes
	apiV1.HandleFunc("/deliveries", GetAllDeliveries).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery/{ordernumber}", GetDelivery).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery", AddDelivery).Methods(http.MethodPost)
	apiV1.HandleFunc("/delivery/{ordernumber}", UpdateDelivery).Methods(http.MethodPut)
	apiV1.HandleFunc("/delivery/{ordernumber}", DeleteDelivery).Methods(http.MethodDelete)
	return router
}

And here is main.go where the DeliveryDB is initialized:

package main

import (
	"log"
	"net/http"
)

var deliverydb DeliveryDB

func main() {
	// Create a database
	deliverydb = NewListDatabase()
	// Start http server
	router := MakeRouter()
	log.Fatal(http.ListenAndServe(":8000", router))
}

Postgres

There are lots of databases out there but for the purposes of this tutorial I chose postgres. It’s essentially the only one out of the traditional RDBMS systems worth using. The Go standard library is pretty rich so there is a database/sql package although we have to provide a database driver to it. The two most popular drivers for postgres are pq and pgx. Since pq is in maintenance mode and pgx is the recommended alternative by the pq authors we’ll go with that. The guide on the pgx wiki is a great starting point but we’ll add a big more to it.

Connection code

Enough rambling, let’s see how we can start using postgres. Create a file called postgres.go with the following contents:

package main

import (
	"database/sql"
	_ "github.com/jackc/pgx/v4/stdlib"
)

type postgresDB struct {
	client *sql.DB
	pgURL  string
}

func newPostgresClient(url string) (*sql.DB, error) {
	client, err := sql.Open("pgx", url)
	if err != nil {
		return nil, err
	}
	err = client.Ping()
	if err != nil {
		return nil, err
	}
	return client, nil
}

func NewPostgresDB(url string) (*postgresDB, error) {
	pgclient, err := newPostgresClient(url)
	if err != nil {
		return nil, err
	}
	db := &postgresDB{
		pgURL:  url,
		client: pgclient,
	}
	return db, nil
}

func (pdb *postgresDB) ReturnAll() ([]*Delivery, error) {
	return []*Delivery{}, nil
}

func (pdb *postgresDB) ReturnOne(orderNumber int) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Store(d *Delivery) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Change(orderNumber int, del *Delivery) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Remove(orderNumber int) error {
	return nil
}

Quite a few stuff going on. First up, the imports. As discussed we can use database/sql from the standard library and then combine it with a driver. The line _ "github.com/jackc/pgx/v4/stdlib" means that we’re importing that dependency but we won’t have to write stdlib.Something to use its functions and variables but instead just write Something. Following that we have the postgresDB struct which will implement the DeliveryDB interface and which stores the database URL and the database client that uses that URL. Next up, there is the initialization code which is split into two functions, one private and one public as indicated by the lowercase/uppercase letter. The private function, named newPostgresClient creates a client for the database through sql.Open using the pgx driver and then pings the database to make sure it’s accessible. The public function runs newPostgresClient with the provided URL and if there are no errors it saves the URL as well as the client to a new instance of postgresDB and returns it to the caller. Last but not least we see that ReturnAll, ReturnOne, Store, Change and Remove methods are implemented on the postgresDB struct as indicated by (pdb *postgresDB). Right now they don’t really do anything, they’re there just so the DeliveryDB interface is satisfied and the compiler doesn’t exit with an error.

Choose database

Now that we have written code to connect to postgres, we should probably set it up in main.go so if the environment variable PG_URL is set, we use that connection string to connect to postgres.

package main

import (
	"log"
	"net/http"
	"os"
)

var deliverydb DeliveryDB

func main() {
	// Create a database
	pgURL := os.Getenv("PG_URL")
	if pgURL != "" {
		db, err := NewPostgresDB(pgURL)
		if err != nil {
			log.Fatalf("error with postgres connection %v", err)
		}
		deliverydb = db
		log.Println("connected to postgres")
	} else {
		deliverydb = NewListDatabase()
		log.Println("connected to listdb")
	}
	// Start http server
	router := MakeRouter()
	log.Fatal(http.ListenAndServe(":8000", router))
}

It’s pretty simple, we get the value of the PG_URL variable and if it’s not empty, we use it to connect to postgres. If it’s empty we just use the fake list database instead so our web service can still run.

Test connection

While we are confident that we did everything right, it doesn’t hurt to test it out. Using docker we can quickly create a test database without permanently storing any data. The following command should bring up a database we can use for testing:

`1`	`docker run -d --rm --name testpostgres -p 5432:5432 -e POSTGRES_PASSWORD=Apasswd -e POSTGRES_USER=tester postgres:latest`

After it’s done, set the PG_URL environment variable like so:

`1`	`export PG_URL="postgresql://tester:Apasswd@localhost:5432?sslmode=disable"`

And then run the service the same way we’ve been doing all along:

`1`	`go run *.go`

The message connected to postgres should appear in the command line. And there we go, our connection is working and we can move on. Kill the running database with:

`1`	`docker stop testpostgres`

And let’s see how to write some SQL queries.

Queries

In order to work with the database we will need to write queries to be used. For the sake of simplicity we won’t deduplicate the delivery drivers but that is something you could do in case you’re looking for ways to tweak the project on your own. With that in mind, what queries are we going to need? One per method of the delivery database interface should be enough, along with one to create the table on first run.

Create table query

Let’s begin then by creating a file called queries.go with the create table query:

package main

var createDeliveriesTableQuery = `CREATE TABLE IF NOT EXISTS Deliveries (
	OrderNumber SERIAL PRIMARY KEY,
	City VARCHAR,
	Zipcode VARCHAR,
	Address VARCHAR,
	Phone1 VARCHAR,
	Phone2 VARCHAR,
	Cancelled BOOL,
	Delivered BOOL,
	DeliveryAttempts INTEGER,
	DriverFirstName VARCHAR,
	DriverLastName VARCHAR
);`

This is an SQL representation of our Delivery model. One thing you might not have seen is SERIAL PRIMARY KEY. It will automatically increment when a new entry is inserted so we don’t have to keep track of it ourselves. Let’s incorporate this query into the postgres.go code:

package main

import (
	"database/sql"
	_ "github.com/jackc/pgx/v4/stdlib"
)

type postgresDB struct {
	client *sql.DB
	pgURL  string
}

func newPostgresClient(url string) (*sql.DB, error) {
	client, err := sql.Open("pgx", url)
	if err != nil {
		return nil, err
	}
	err = client.Ping()
	if err != nil {
		return nil, err
	}
	_, err = client.Exec(createDeliveriesTableQuery)
	if err != nil {
		return nil, err
	}
	return client, nil
}

func NewPostgresDB(url string) (*postgresDB, error) {
	pgclient, err := newPostgresClient(url)
	if err != nil {
		return nil, err
	}
	db := &postgresDB{
		pgURL:  url,
		client: pgclient,
	}
	return db, nil
}

func (pdb *postgresDB) ReturnAll() ([]*Delivery, error) {
	return []*Delivery{}, nil
}

func (pdb *postgresDB) ReturnOne(orderNumber int) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Store(d *Delivery) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Change(orderNumber int, del *Delivery) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Remove(orderNumber int) error {
	return nil
}

Just 4 new lines and our database will be set up on first connection. You can run the connection test again as we did in the previous section.

See all deliveries

This requires a simple query, we just select everything in the Deliveries table so take a loot at queries.go:

package main

var createDeliveriesTableQuery = `CREATE TABLE IF NOT EXISTS Deliveries (
	OrderNumber SERIAL PRIMARY KEY,
	City VARCHAR,
	Zipcode VARCHAR,
	Address VARCHAR,
	Phone1 VARCHAR,
	Phone2 VARCHAR,
	Cancelled BOOL,
	Delivered BOOL,
	DeliveryAttempts INTEGER,
	DriverFirstName VARCHAR,
	DriverLastName VARCHAR
);`
var retrieveAllDeliveriesQuery = `SELECT 
	OrderNumber,
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName FROM Deliveries;`

And then we’ll use it in the ReturnAll method inside postgres.go

package main

import (
	"database/sql"
	_ "github.com/jackc/pgx/v4/stdlib"
)

type postgresDB struct {
	client *sql.DB
	pgURL  string
}

func newPostgresClient(url string) (*sql.DB, error) {
	client, err := sql.Open("pgx", url)
	if err != nil {
		return nil, err
	}
	err = client.Ping()
	if err != nil {
		return nil, err
	}
	_, err = client.Exec(createDeliveriesTableQuery)
	if err != nil {
		return nil, err
	}
	return client, nil
}

func NewPostgresDB(url string) (*postgresDB, error) {
	pgclient, err := newPostgresClient(url)
	if err != nil {
		return nil, err
	}
	db := &postgresDB{
		pgURL:  url,
		client: pgclient,
	}
	return db, nil
}

func (pdb *postgresDB) ReturnAll() ([]*Delivery, error) {
	var deliveryslice = []*Delivery{}
	rows, err := pdb.client.Query(retrieveAllDeliveriesQuery)
	if err != nil {
		return nil, err
	}
	defer rows.Close()
	for rows.Next() {
		var delivery = &Delivery{}
		err = rows.Scan(
			&delivery.OrderNumber,
			&delivery.City,
			&delivery.Zipcode,
			&delivery.Address,
			&delivery.Phone1,
			&delivery.Phone2,
			&delivery.Cancelled,
			&delivery.Delivered,
			&delivery.DeliveryAttempts,
			&delivery.Driver.FirstName,
			&delivery.Driver.LastName,
		)
		if err != nil {
			return nil, err
		}
		deliveryslice = append(deliveryslice, delivery)
	}
	return deliveryslice, nil
}

func (pdb *postgresDB) ReturnOne(orderNumber int) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Store(d *Delivery) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Change(orderNumber int, del *Delivery) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Remove(orderNumber int) error {
	return nil
}

Initially, we create an empty list of deliveries to store all the deliveries retrieved from the database. Then we run the query, iterate over the rows returned, save the results to the delivery struct and append that struct to the list. Finally we return the list to the caller and that’s all.

See one delivery

This one is also pretty simple, we select everything if the entry has the specified order number:

package main

var createDeliveriesTableQuery = `CREATE TABLE IF NOT EXISTS Deliveries (
	OrderNumber SERIAL PRIMARY KEY,
	City VARCHAR,
	Zipcode VARCHAR,
	Address VARCHAR,
	Phone1 VARCHAR,
	Phone2 VARCHAR,
	Cancelled BOOL,
	Delivered BOOL,
	DeliveryAttempts INTEGER,
	DriverFirstName VARCHAR,
	DriverLastName VARCHAR
);`
var retrieveAllDeliveriesQuery = `SELECT 
	OrderNumber,
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName FROM Deliveries;`
var retrieveOneDeliveryQuery = `SELECT
	OrderNumber,
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName
FROM Deliveries WHERE OrderNumber=$1;`

And now we need to use it inside our ReturnOne method:

package main

import (
	"database/sql"
	_ "github.com/jackc/pgx/v4/stdlib"
)

type postgresDB struct {
	client *sql.DB
	pgURL  string
}

func newPostgresClient(url string) (*sql.DB, error) {
	client, err := sql.Open("pgx", url)
	if err != nil {
		return nil, err
	}
	err = client.Ping()
	if err != nil {
		return nil, err
	}
	_, err = client.Exec(createDeliveriesTableQuery)
	if err != nil {
		return nil, err
	}
	return client, nil
}

func NewPostgresDB(url string) (*postgresDB, error) {
	pgclient, err := newPostgresClient(url)
	if err != nil {
		return nil, err
	}
	db := &postgresDB{
		pgURL:  url,
		client: pgclient,
	}
	return db, nil
}

func (pdb *postgresDB) ReturnAll() ([]*Delivery, error) {
	var deliveryslice = []*Delivery{}
	rows, err := pdb.client.Query(retrieveAllDeliveriesQuery)
	if err != nil {
		return nil, err
	}
	defer rows.Close()
	for rows.Next() {
		var delivery = &Delivery{}
		err = rows.Scan(
			&delivery.OrderNumber,
			&delivery.City,
			&delivery.Zipcode,
			&delivery.Address,
			&delivery.Phone1,
			&delivery.Phone2,
			&delivery.Cancelled,
			&delivery.Delivered,
			&delivery.DeliveryAttempts,
			&delivery.Driver.FirstName,
			&delivery.Driver.LastName,
		)
		if err != nil {
			return nil, err
		}
		deliveryslice = append(deliveryslice, delivery)
	}
	return deliveryslice, nil
}

func (pdb *postgresDB) ReturnOne(orderNumber int) (*Delivery, error) {
	var delivery = &Delivery{}
	err := pdb.client.QueryRow(retrieveOneDeliveryQuery, orderNumber).Scan(
		&delivery.OrderNumber,
		&delivery.City,
		&delivery.Zipcode,
		&delivery.Address,
		&delivery.Phone1,
		&delivery.Phone2,
		&delivery.Cancelled,
		&delivery.Delivered,
		&delivery.DeliveryAttempts,
		&delivery.Driver.FirstName,
		&delivery.Driver.LastName,
	)
	if err != nil {
		return nil, err
	}
	return delivery, nil
}

func (pdb *postgresDB) Store(d *Delivery) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Change(orderNumber int, del *Delivery) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Remove(orderNumber int) error {
	return nil
}

This is basically the same as ReturnAll but even simpler so I won’t cover it in detail.

Add one delivery

In order to test the retrieve methods, we need to have a way to store stuff first. Since the database generates the order number, the query needs to return it when we run it.

package main

var createDeliveriesTableQuery = `CREATE TABLE IF NOT EXISTS Deliveries (
	OrderNumber SERIAL PRIMARY KEY,
	City VARCHAR,
	Zipcode VARCHAR,
	Address VARCHAR,
	Phone1 VARCHAR,
	Phone2 VARCHAR,
	Cancelled BOOL,
	Delivered BOOL,
	DeliveryAttempts INTEGER,
	DriverFirstName VARCHAR,
	DriverLastName VARCHAR
);`
var retrieveAllDeliveriesQuery = `SELECT 
	OrderNumber,
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName FROM Deliveries;`
var retrieveOneDeliveryQuery = `SELECT
	OrderNumber,
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName
FROM Deliveries WHERE OrderNumber=$1;`
var storeDeliveryQuery = `INSERT INTO Deliveries (
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) RETURNING OrderNumber;`

This way when we insert an entry, we will get back its order number. Let’s see how the Store method will be implemented

package main

import (
	"database/sql"
	_ "github.com/jackc/pgx/v4/stdlib"
)

type postgresDB struct {
	client *sql.DB
	pgURL  string
}

func newPostgresClient(url string) (*sql.DB, error) {
	client, err := sql.Open("pgx", url)
	if err != nil {
		return nil, err
	}
	err = client.Ping()
	if err != nil {
		return nil, err
	}
	_, err = client.Exec(createDeliveriesTableQuery)
	if err != nil {
		return nil, err
	}
	return client, nil
}

func NewPostgresDB(url string) (*postgresDB, error) {
	pgclient, err := newPostgresClient(url)
	if err != nil {
		return nil, err
	}
	db := &postgresDB{
		pgURL:  url,
		client: pgclient,
	}
	return db, nil
}

func (pdb *postgresDB) ReturnAll() ([]*Delivery, error) {
	var deliveryslice = []*Delivery{}
	rows, err := pdb.client.Query(retrieveAllDeliveriesQuery)
	if err != nil {
		return nil, err
	}
	defer rows.Close()
	for rows.Next() {
		var delivery = &Delivery{}
		err = rows.Scan(
			&delivery.OrderNumber,
			&delivery.City,
			&delivery.Zipcode,
			&delivery.Address,
			&delivery.Phone1,
			&delivery.Phone2,
			&delivery.Cancelled,
			&delivery.Delivered,
			&delivery.DeliveryAttempts,
			&delivery.Driver.FirstName,
			&delivery.Driver.LastName,
		)
		if err != nil {
			return nil, err
		}
		deliveryslice = append(deliveryslice, delivery)
	}
	return deliveryslice, nil
}

func (pdb *postgresDB) ReturnOne(orderNumber int) (*Delivery, error) {
	var delivery = &Delivery{}
	err := pdb.client.QueryRow(retrieveOneDeliveryQuery, orderNumber).Scan(
		&delivery.OrderNumber,
		&delivery.City,
		&delivery.Zipcode,
		&delivery.Address,
		&delivery.Phone1,
		&delivery.Phone2,
		&delivery.Cancelled,
		&delivery.Delivered,
		&delivery.DeliveryAttempts,
		&delivery.Driver.FirstName,
		&delivery.Driver.LastName,
	)
	if err != nil {
		return nil, err
	}
	return delivery, nil
}

func (pdb *postgresDB) Store(d *Delivery) (*Delivery, error) {
	err := pdb.client.QueryRow(storeDeliveryQuery,
		d.City,
		d.Zipcode,
		d.Address,
		d.Phone1,
		d.Phone2,
		d.Cancelled,
		d.Delivered,
		d.DeliveryAttempts,
		d.Driver.FirstName,
		d.Driver.LastName,
	).Scan(&d.OrderNumber)
	if err != nil {
		return nil, err
	}
	return d, nil
}

func (pdb *postgresDB) Change(orderNumber int, del *Delivery) (*Delivery, error) {
	return &Delivery{}, nil
}

func (pdb *postgresDB) Remove(orderNumber int) error {
	return nil
}

In this method we just insert what the user sent, ignoring the order number they specified, and then return the entry as it was saved.

Update one delivery

There are a few ways to go about updating existing entries. We can check each thing about the Delivery that is sent to us and update only the things that are changed. Another alternative is to swap out all existing values with what is sent which is the dumb method. The first option allows for sending less data and maybe saving some bandwidth but the second one is easier to implement so we’ll hope whoever uses our API does so responsibly. Below is queries.go with our update query added. Note that I used some different SQL syntax this time to keep it a little interesting:

package main

var createDeliveriesTableQuery = `CREATE TABLE IF NOT EXISTS Deliveries (
	OrderNumber SERIAL PRIMARY KEY,
	City VARCHAR,
	Zipcode VARCHAR,
	Address VARCHAR,
	Phone1 VARCHAR,
	Phone2 VARCHAR,
	Cancelled BOOL,
	Delivered BOOL,
	DeliveryAttempts INTEGER,
	DriverFirstName VARCHAR,
	DriverLastName VARCHAR
);`
var retrieveAllDeliveriesQuery = `SELECT 
	OrderNumber,
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName FROM Deliveries;`
var retrieveOneDeliveryQuery = `SELECT
	OrderNumber,
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName
FROM Deliveries WHERE OrderNumber=$1;`
var storeDeliveryQuery = `INSERT INTO Deliveries (
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) RETURNING OrderNumber;`
var changeDeliveryQuery = `UPDATE Deliveries SET
	City=$2,
	Zipcode=$3,
	Address=$4,
	Phone1=$5,
	Phone2=$6,
	Cancelled=$7,
	Delivered=$8,
	DeliveryAttempts=$9,
	DriverFirstName=$10
	DriverLastName=$11
WHERE OrderNumber=$1;`

Not too bad I’d say, we just set the values given if the entry has the order number given. Using this in postgres is not too hard since we’ve seen the previous ones, give it a look:

package main

import (
	"database/sql"
	_ "github.com/jackc/pgx/v4/stdlib"
)

type postgresDB struct {
	client *sql.DB
	pgURL  string
}

func newPostgresClient(url string) (*sql.DB, error) {
	client, err := sql.Open("pgx", url)
	if err != nil {
		return nil, err
	}
	err = client.Ping()
	if err != nil {
		return nil, err
	}
	_, err = client.Exec(createDeliveriesTableQuery)
	if err != nil {
		return nil, err
	}
	return client, nil
}

func NewPostgresDB(url string) (*postgresDB, error) {
	pgclient, err := newPostgresClient(url)
	if err != nil {
		return nil, err
	}
	db := &postgresDB{
		pgURL:  url,
		client: pgclient,
	}
	return db, nil
}

func (pdb *postgresDB) ReturnAll() ([]*Delivery, error) {
	var deliveryslice = []*Delivery{}
	rows, err := pdb.client.Query(retrieveAllDeliveriesQuery)
	if err != nil {
		return nil, err
	}
	defer rows.Close()
	for rows.Next() {
		var delivery = &Delivery{}
		err = rows.Scan(
			&delivery.OrderNumber,
			&delivery.City,
			&delivery.Zipcode,
			&delivery.Address,
			&delivery.Phone1,
			&delivery.Phone2,
			&delivery.Cancelled,
			&delivery.Delivered,
			&delivery.DeliveryAttempts,
			&delivery.Driver.FirstName,
			&delivery.Driver.LastName,
		)
		if err != nil {
			return nil, err
		}
		deliveryslice = append(deliveryslice, delivery)
	}
	return deliveryslice, nil
}

func (pdb *postgresDB) ReturnOne(orderNumber int) (*Delivery, error) {
	var delivery = &Delivery{}
	err := pdb.client.QueryRow(retrieveOneDeliveryQuery, orderNumber).Scan(
		&delivery.OrderNumber,
		&delivery.City,
		&delivery.Zipcode,
		&delivery.Address,
		&delivery.Phone1,
		&delivery.Phone2,
		&delivery.Cancelled,
		&delivery.Delivered,
		&delivery.DeliveryAttempts,
		&delivery.Driver.FirstName,
		&delivery.Driver.LastName,
	)
	if err != nil {
		return nil, err
	}
	return delivery, nil
}

func (pdb *postgresDB) Store(d *Delivery) (*Delivery, error) {
	err := pdb.client.QueryRow(storeDeliveryQuery,
		d.City,
		d.Zipcode,
		d.Address,
		d.Phone1,
		d.Phone2,
		d.Cancelled,
		d.Delivered,
		d.DeliveryAttempts,
		d.Driver.FirstName,
		d.Driver.LastName,
	).Scan(&d.OrderNumber)
	if err != nil {
		return nil, err
	}
	return d, nil
}

func (pdb *postgresDB) Change(orderNumber int, del *Delivery) (*Delivery, error) {
	_, err := pdb.client.Exec(changeDeliveryQuery,
		orderNumber,
		del.City,
		del.Zipcode,
		del.Address,
		del.Phone1,
		del.Phone2,
		del.Cancelled,
		del.Delivered,
		del.DeliveryAttempts,
		del.Driver.FirstName,
		del.Driver.LastName,
	)
	if err != nil {
		return nil, err
	}
	return pdb.ReturnOne(orderNumber)
}

func (pdb *postgresDB) Remove(orderNumber int) error {
	return nil
}

As you can see we don’t use the order number from the delivery object since the user might have maliciously changed it. In addition, ReturnOne is called to make sure that we send back the changed entry as it is in the database so there is no data discrepancies between what we store and what we show.

Delete one delivery

If you remember, we don’t actually delete deliveries, we just set the Cancelled status to true. However for the sake of showing how the query would be, I’ll include another one along with its usage inside prostgres as comments alongside the proper one.

package main

var createDeliveriesTableQuery = `CREATE TABLE IF NOT EXISTS Deliveries (
	OrderNumber SERIAL PRIMARY KEY,
	City VARCHAR,
	Zipcode VARCHAR,
	Address VARCHAR,
	Phone1 VARCHAR,
	Phone2 VARCHAR,
	Cancelled BOOL,
	Delivered BOOL,
	DeliveryAttempts INTEGER,
	DriverFirstName VARCHAR,
	DriverLastName VARCHAR
);`
var retrieveAllDeliveriesQuery = `SELECT 
	OrderNumber,
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName FROM Deliveries;`
var retrieveOneDeliveryQuery = `SELECT
	OrderNumber,
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName
FROM Deliveries WHERE OrderNumber=$1;`
var storeDeliveryQuery = `INSERT INTO Deliveries (
	City,
	Zipcode,
	Address,
	Phone1,
	Phone2,
	Cancelled,
	Delivered,
	DeliveryAttempts,
	DriverFirstName,
	DriverLastName
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) RETURNING OrderNumber;`
var changeDeliveryQuery = `UPDATE Deliveries SET
	City=$2,
	Zipcode=$3,
	Address=$4,
	Phone1=$5,
	Phone2=$6,
	Cancelled=$7,
	Delivered=$8,
	DeliveryAttempts=$9,
	DriverFirstName=$10
	DriverLastName=$11
WHERE OrderNumber=$1;`
var removeDeliveryQuery = `UPDATE Deliveries SET Cancelled=true WHERE OrderNumber=$1`
//commented out so the compiler doesn't complain about unused variables
//var removeDeliveryForRealQuery = `DELETE FROM Deliveries WHERE OrderNumber=$1`

The query we’ll use is another update, this time it simply sets the Cancelled field to true. Here is the postgres part of it:

package main

import (
	"database/sql"
	_ "github.com/jackc/pgx/v4/stdlib"
)

type postgresDB struct {
	client *sql.DB
	pgURL  string
}

func newPostgresClient(url string) (*sql.DB, error) {
	client, err := sql.Open("pgx", url)
	if err != nil {
		return nil, err
	}
	err = client.Ping()
	if err != nil {
		return nil, err
	}
	_, err = client.Exec(createDeliveriesTableQuery)
	if err != nil {
		return nil, err
	}
	return client, nil
}

func NewPostgresDB(url string) (*postgresDB, error) {
	pgclient, err := newPostgresClient(url)
	if err != nil {
		return nil, err
	}
	db := &postgresDB{
		pgURL:  url,
		client: pgclient,
	}
	return db, nil
}

func (pdb *postgresDB) ReturnAll() ([]*Delivery, error) {
	var deliveryslice = []*Delivery{}
	rows, err := pdb.client.Query(retrieveAllDeliveriesQuery)
	if err != nil {
		return nil, err
	}
	defer rows.Close()
	for rows.Next() {
		var delivery = &Delivery{}
		err = rows.Scan(
			&delivery.OrderNumber,
			&delivery.City,
			&delivery.Zipcode,
			&delivery.Address,
			&delivery.Phone1,
			&delivery.Phone2,
			&delivery.Cancelled,
			&delivery.Delivered,
			&delivery.DeliveryAttempts,
			&delivery.Driver.FirstName,
			&delivery.Driver.LastName,
		)
		if err != nil {
			return nil, err
		}
		deliveryslice = append(deliveryslice, delivery)
	}
	return deliveryslice, nil
}

func (pdb *postgresDB) ReturnOne(orderNumber int) (*Delivery, error) {
	var delivery = &Delivery{}
	err := pdb.client.QueryRow(retrieveOneDeliveryQuery, orderNumber).Scan(
		&delivery.OrderNumber,
		&delivery.City,
		&delivery.Zipcode,
		&delivery.Address,
		&delivery.Phone1,
		&delivery.Phone2,
		&delivery.Cancelled,
		&delivery.Delivered,
		&delivery.DeliveryAttempts,
		&delivery.Driver.FirstName,
		&delivery.Driver.LastName,
	)
	if err != nil {
		return nil, err
	}
	return delivery, nil
}

func (pdb *postgresDB) Store(d *Delivery) (*Delivery, error) {
	err := pdb.client.QueryRow(storeDeliveryQuery,
		d.City,
		d.Zipcode,
		d.Address,
		d.Phone1,
		d.Phone2,
		d.Cancelled,
		d.Delivered,
		d.DeliveryAttempts,
		d.Driver.FirstName,
		d.Driver.LastName,
	).Scan(&d.OrderNumber)
	if err != nil {
		return nil, err
	}
	return d, nil
}

func (pdb *postgresDB) Change(orderNumber int, del *Delivery) (*Delivery, error) {
	_, err := pdb.client.Exec(changeDeliveryQuery,
		orderNumber,
		del.City,
		del.Zipcode,
		del.Address,
		del.Phone1,
		del.Phone2,
		del.Cancelled,
		del.Delivered,
		del.DeliveryAttempts,
		del.Driver.FirstName,
		del.Driver.LastName,
	)
	if err != nil {
		return nil, err
	}
	return pdb.ReturnOne(orderNumber)
}

func (pdb *postgresDB) Remove(orderNumber int) error {
	_, err := pdb.client.Exec(removeDeliveryQuery, orderNumber)
	if err != nil {
		return err
	}
	return nil
}

Just pass the order number and execute the query. Simple enough.

Run last test

With the code completed, run postgres using the docker command:

`1`	`docker run -d --rm --name testpostgres -p 5432:5432 -e POSTGRES_PASSWORD=Apasswd -e POSTGRES_USER=tester postgres:latest`

and then run the following commands to test all functionality:

curl http://localhost:8000/api/v1/deliveries
curl -X POST -d '{"ordernumber":2,"city":"There","zipcode":"1701","address":"Office","phone1":"6932728091","cancelled":false,"delivered":true,"deliveryattempts":1,"deliverydriver":{"firstname":"Lucas","lastname":"Johnson"}}' http://localhost/api/v1/delivery
curl http://localhost:8000/api/v1/deliveries
curl -X UPDATE -d '{"ordernumber":97,"city":"Here","zipcode":"1701","address":"Home Office","phone1":"6932728091","cancelled":false,"delivered":true,"deliveryattempts":1,"deliverydriver":{"firstname":"Lucas","lastname":"Johnson"}}' http://localhost/api/v1/delivery
curl http://localhost:8000/api/v1/deliveries
curl -X DELETE http://localhost/api/v1/delivery/1
curl http://localhost:8000/api/v1/deliveries

Ta-da! We now have a web service backed by a database. It’s sophisticated enough to do the basic CRUD operations and it’s a fairly relatable domain; deliveries.

Conclusion

That was a long one, wasn’t it? It took forever to write, about a month of working on it on and off. Tried my best to give a complete picture and go through the process step by step, let me know how you think that went. Next time I’d like to cover testing but I’m not sure when that part will come out. Until then, I hope you learned something.

The Power of the Library Example

Thu, 06 May 2021 14:25:41 +0300

While considering what kind of example I could use for a service that was supposed to be a tech demo, I settled on a catalog/library of books. I think it’s a powerful example because it includes the basic CRUD operations and also allows for a lot of flexibility and detail of implementation. Below I’ll try to explain the most important ones and maybe help convince you to rethink about it and maybe use it as the next step after the usual todo list one.

The Library example

Let’s start with an explanation of what this example actually is before going into its advantages. It’s pretty simple, you have a library of books and want to perform operations associated with managing a library. Simple concept, simple to implement and uses real-life things that people have experience with and can reason about. Is it really this simple though? In the intro I said it’s supposed to be a step-up from the usual todo list example but so far it doesn’t seem different other than being another object in the place of a todo item. Well, let’s dive a little deeper.

Data Structures

When you start, there is the dilemma of what fields/attributes to give the book. The ISBN stands out as a convenient unique identifier which is great, then there is the title, the number of pages, year of release, format as in hard cover or soft cover or pdf or epub or mobi, edition, categor(y/ies), author, publisher, subtitle, cover art, bookmark for reading progress, description in the back cover, if you own it or not, did you lend it to someone or not, are you lending it from someone or not, if it’s a physical copy where is it located and so on and so forth. I think you get my point, you can get really deep into it if you want or pick some of the basics and go from there. Are they really basics though? Specifically when it comes to who wrote the book, it could be one person or it could be more than one. Especially in technical books it’s not unusual to have 2 or 3 authors so you’re getting into a situation where you have something like the following (silly example but please stay with me):

{
  "books": [
    {
      "ISBN": "0189219181",
      "Title": "hey",
      "Author": {
        "authors": [
          {
            "AuthorID": 1,
            "FirstName": "anguish",
            "MiddleName": "none",
            "LastName": "mental",
            "YearBorn": 404,
            "YearDied": 201,
            "BooksWritten": 5
          },
          {
            "AuthorID": 2,
            "FirstName": "big",
            "MiddleName": "boo",
            "LastName": "ba",
            "BooksWritten": 9543
          },
          {
            "AuthorID": 3,
            "FirstName": "me",
            "MiddleName": "notme",
            "LastName": "notyou",
            "YearBorn": 1999,
            "BooksWritten": 2
          }
        ]
      },
      "Year": 1999,
      "Edition": 1,
      "Publisher": {
        "PublisherID": 3,
        "Name": "urmom",
        "YearStarted": 1237,
        "YearEnded": 2077,
        "BooksPublished": 9001
      },
      "Pages": 100,
      "Category": "tech",
      "PDF": false,
      "Owned": true,
      "Lended": true
    }
  ]
}

Here we see that each author is inside an array which is the value of a key called authors which is the value of a key called Author which is a part of one of the objects in the array called books which is inside the outer JSON object. Fairly complex, right? Additionally, you have to pay attention to this throughout the entire application including saving it and retrieving it from a database, when passing it to other services that might use this data for example a cover art finder or a media aggregator service and when those services process it. Following that, there are logistical considerations about storing an author or publisher twice since the client shouldn’t have to deal with sending the right ID for the publisher or the author(s), checking if a book with the same ISBN exists, potentially handling cases where you bought it again cause the person you gave it to never gave it back and so on. The main takeaway is that you can decide how real-world you want to make the example, add complexity at will and introduce edge cases that defy previously established preconditions such as the ISBN being a unique identifier.

Functions

I touched on this before I believe it warrants repeating. Similar to the data structure you go with, there several functions of differing depth you can implement. The simple CRUD operations might be taken as a baseline but what even if that rule was able to be defied? Do you want to delete the book or simply set a hidden state because you want the library to act as an archive? Should the cases of the author’s year of birth and death be allowed to be set incorrectly (like in the example above with author 1), partially (author 2) or not at all (author 3)? Should the person that the library belongs to be able to mark the books as lended but not necessarily to who or should they be able to record who they lended the books to? Are the books editable after being added or not? Should there be a url for the digital edition visible to anyone or only a signed-in and authenticated person? Do we even want to support multiple people using the same instance of this library service? So many different things to consider and this can be a perfect opportunity to learn to use data while also being the one that defines what that data should look like. You get be the one producing and consuming the data which is not the case with 3rd-party APIs thus giving you the opportunity to figure things out on the way there. And speaking of accessing the data…

Interfacing

I’ve only spoken about server-side stuff this far (unless you count the output of jq -M as a frontend) but let’s round this out with the client-side. I showed some example JSON previously but in the intro there was a mention of a tech demo and you can bet I’m not writing REST for this. The main definition in my case is a protobuf file so I get control of the struct only as far as the proto v3 syntax allows. This presents an interesting challenge where there is limited or no option for annotations so no ORM (gorm for golang) can be used. Since I’m using gRPC with that and also want JSON to be able to be returned, a translation layer was used, grpc-gateway to be specific. With the ability to use JSON to communicate, a cli, desktop, mobile or web client can be made in a variety of languages while not losing the great service-to-service communication that gRPC provides. After having mostly nailed down the data structures I wanted to use and most of the entity-related functions I wanted to implement I decided to take a look at reactjs again. A post about it will come later after I’m done with this project but the example is given to demonstrate that you can pick something that can make an http request and process JSON then run off with it and build something. The way you choose access the library is noteworthy and can become another important part of using this concept as the example in a tutorial.

Conclusion

I believe the book/library example to be a very good one as a second step after a todo list one due to the complexity that it allows to be introduced in the data structures, the functions, the interfaces and APIs as well as the possibilities it opens up. Thank you for reading, I hope you enjoyed it and maybe learned something.

Go API Part 2

Tue, 06 Apr 2021 02:23:55 +0300

Previous relevant tutorial

The introduction, setup and explanation of basic concepts and operations was covered in part 1 so make sure to read that first.

Intro

In this part we will be developing a full CRUD REST API in Go except without a real database because I realize that it’s too much to go over at once. The rest of what I promised is true though, we’ll use real data that is more than just a basic todo, explore routing options, subrouters and even middleware.

Starting point

The place we start at is a basic main.go file with our structs that represent the models we’ll work with as well as a simple gorilla/mux setup. The first part of CRUD that we will implement is the simplest one, Read. More specifically reading all the entries. We will also create a slice (essentially a list) of pointers for the instances of Delivery that will exist. The HTTP Method we specify that we accept is http.MethodGet since whoever is accessing the API is getting data from us, not sending us anything. In addition, we set the http status code to http.StatusOK in order to communicate that the request went well.

// main.go
package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
)

type DeliveryDriver struct {
	FirstName string `json:"firstname"`
	LastName  string `json:"lastname"`
}

type Delivery struct {
	OrderNumber      int            `json:"ordernumber"`
	City             string         `json:"city"`
	Zipcode          string         `json:"zipcode"`
	Address          string         `json:"address"`
	Phone1           string         `json:"phone1"`
	Phone2           string         `json:"phone2,omitempty"`
	Cancelled        bool           `json:"cancelled"`
	Delivered        bool           `json:"delivered"`
	DeliveryAttempts int            `json:"deliveryattempts"`
	Driver           DeliveryDriver `json:"deliverydriver"`
}

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{}

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(deliveries)
}

func main() {
	// Set up router
	router := mux.NewRouter()
	// Set up subrouter for api version 1
	apiV1 := router.PathPrefix("/api/v1").Subrouter()
	// Set up routes
	apiV1.HandleFunc("/deliveries", GetAllDeliveries).Methods(http.MethodGet)
	// Start http server
	log.Fatal(http.ListenAndServe(":8000", router))
}

As you can see it’s fairly familiar. The subrouter is based on the path so anything starting with /api/v1 is sent to that subrouter and all the HandleFunc only needs to specify the path under the path of the subrouter, in this case /deliveries. We can run this using go run main.go and visit localhost:8000/api/v1/deliveries to see a measly [] which is an empty json array.

Get all

Let’s add some data then, 3 example entries should suffice:

// main.go
package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
)

type DeliveryDriver struct {
	FirstName string `json:"firstname"`
	LastName  string `json:"lastname"`
}

type Delivery struct {
	OrderNumber      int            `json:"ordernumber"`
	City             string         `json:"city"`
	Zipcode          string         `json:"zipcode"`
	Address          string         `json:"address"`
	Phone1           string         `json:"phone1"`
	Phone2           string         `json:"phone2,omitempty"`
	Cancelled        bool           `json:"cancelled"`
	Delivered        bool           `json:"delivered"`
	DeliveryAttempts int            `json:"deliveryattempts"`
	Driver           DeliveryDriver `json:"deliverydriver"`
}

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{
	&Delivery{
		OrderNumber:      1,
		City:             "Here",
		Zipcode:          "52011",
		Address:          "Home",
		Phone1:           "6945123789",
		Phone2:           "2313722903",
		Cancelled: false,
		Delivered:        false,
		DeliveryAttempts: 0,
		Driver: DeliveryDriver{
			FirstName: "Mhtsos",
			LastName:  "Iwannou",
		},
	},
	&Delivery{
		OrderNumber:      2,
		City:             "There",
		Zipcode:          "1701",
		Address:          "Office",
		Phone1:           "6932728091",
		Cancelled: false,
		Delivered:        true,
		DeliveryAttempts: 1,
		Driver: DeliveryDriver{
			FirstName: "Lucas",
			LastName:  "Johnson",
		},
	},
	&Delivery{
		OrderNumber:      3,
		City:             "FarAway",
		Zipcode:          "920639",
		Address:          "Island",
		Phone1:           "6900777123",
		Cancelled: true,
		Delivered:        false,
		DeliveryAttempts: 24,
		Driver: DeliveryDriver{
			FirstName: "Pilotos",
			LastName:  "Aeroplanou",
		},
	},
}

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(deliveries)
}

func main() {
	// Set up router
	router := mux.NewRouter()
	// Set up subrouter for api version 1
	apiV1 := router.PathPrefix("/api/v1").Subrouter()
	// Set up routes
	apiV1.HandleFunc("/deliveries", GetAllDeliveries).Methods(http.MethodGet)
	// Start http server
	log.Fatal(http.ListenAndServe(":8000", router))
}

Now after running go run main.go and going to localhost:8000/api/v1/deliveries we can see a whole lot more and if you’re using firefox you’ll get some pretty sweet formatting too (that’s due to setting Content-Type to application/json, if you delete that line it won’t format it).

Get one

Seeing all the entires is fine and dandy but we might want to see information about only one of them. In contrast to the previous one where we used /api/v1/deliveries which means many (in fact it means all because we didn’t specify a range), here we’re going to do something different. We’ll use /api/v1/delivery/ordernumber (notice that delivery is singular) and we’ll specify the ordernumber since it is unique as seen from the example entries. As for the code within the function, we’ll check the parameters to make sure that the argument is actually a number we can use to find the delivery and if it’s fine we will return the delivery as JSON as well as set the status code to http.StatusOK. However if it’s wrong, we’ll set the status code to http.StatusNotFound and then return an error message.

// main.go
package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
	"strconv"
)

type DeliveryDriver struct {
	FirstName string `json:"firstname"`
	LastName  string `json:"lastname"`
}

type Delivery struct {
	OrderNumber      int            `json:"ordernumber"`
	City             string         `json:"city"`
	Zipcode          string         `json:"zipcode"`
	Address          string         `json:"address"`
	Phone1           string         `json:"phone1"`
	Phone2           string         `json:"phone2,omitempty"`
	Cancelled        bool           `json:"cancelled"`
	Delivered        bool           `json:"delivered"`
	DeliveryAttempts int            `json:"deliveryattempts"`
	Driver           DeliveryDriver `json:"deliverydriver"`
}

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{
	&Delivery{
		OrderNumber:      1,
		City:             "Here",
		Zipcode:          "52011",
		Address:          "Home",
		Phone1:           "6945123789",
		Phone2:           "2313722903",
		Cancelled: false,
		Delivered:        false,
		DeliveryAttempts: 0,
		Driver: DeliveryDriver{
			FirstName: "Mhtsos",
			LastName:  "Iwannou",
		},
	},
	&Delivery{
		OrderNumber:      2,
		City:             "There",
		Zipcode:          "1701",
		Address:          "Office",
		Phone1:           "6932728091",
		Cancelled: false,
		Delivered:        true,
		DeliveryAttempts: 1,
		Driver: DeliveryDriver{
			FirstName: "Lucas",
			LastName:  "Johnson",
		},
	},
	&Delivery{
		OrderNumber:      3,
		City:             "FarAway",
		Zipcode:          "920639",
		Address:          "Island",
		Phone1:           "6900777123",
		Cancelled: true,
		Delivered:        false,
		DeliveryAttempts: 24,
		Driver: DeliveryDriver{
			FirstName: "Pilotos",
			LastName:  "Aeroplanou",
		},
	},
}

// Constant for Bad Request
const BadReq string = `{"error": "bad request"}`
// Constant for Not Found
const NotFound string = `{"error": "not found"}`

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(deliveries)
}

// Read a specific delivery
func GetDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

func main() {
	// Set up router
	router := mux.NewRouter()
	// Set up subrouter for api version 1
	apiV1 := router.PathPrefix("/api/v1").Subrouter()
	// Set up routes
	apiV1.HandleFunc("/deliveries", GetAllDeliveries).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery/{ordernumber}", GetDelivery).Methods(http.MethodGet)
	// Start http server
	log.Fatal(http.ListenAndServe(":8000", router))
}

I went ahead and made the errors const variables in the global scope so we can use them anywhere we want without having to repeat ourselves, this is not necessary but it makes life a bit easier. Let’s look at a couple things that seem a bit odd. First what’s that mux.Vars(r)? That’s from gorilla/mux and one of the reasons we are using it. It takes a request as an argument and can tell us a lot of stuff about that request. The variable params is a type map[string]string which means that the key is a string and the value is a string. If you’ve used dictionaries in Python I think it’s pretty similar. We give it the ordernumber key which is what we specified in apiV1.HandleFunc("/delivery/{ordernumber}", GetDelivery).Methods(http.MethodGet) and then use the strconv library to convert it from string to an integer. Second, the for loop looks a bit weird, what’s up with that range deliveries? Using range we can iterate over something, in this case a slice (essentially a list). This returns two variables, in this case one that is the key/index which we are ignoring by using _ and the other one is the value d which is a single delivery from the list of deliveries we set up. With all that out of the way, time to test it. Run the server with go run main.go and visit localhost:8000/api/v1/deliveries/2 to see the information about the second delivery returned. Pretty good, right?

Add one

After we’re done with the Read part of CRUD, let’s move on to Create. Here we will add a delivery after it is sent to us as JSON. I will use cURL for this purpose but you can use something else if you want. Let’s look at the code first though. In our addDelivery function we will create an empty delivery item, d, that will later on receive the JSON data. Then we’ll see what the next order number should be according to the length of our list and use that instead of whatever the user has provided, it is up to us to decide where to fit the delivery. After than, we unpack the contents of the request body into d and check for errors, if there are any, we will handle it like previously since the sender made a mistake. If the ordernumber in their JSON is empty, we don’t mind since we’ll override it before saving it anyway. Following that, we will use append to add the delivery to the list and then set the HTTP status code to http.StatusCreated. Finally we will return the item that we actually inserted into our list of deliveries so the sender knows what was actually saved.

// main.go
package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
	"strconv"
)

type DeliveryDriver struct {
	FirstName string `json:"firstname"`
	LastName  string `json:"lastname"`
}

type Delivery struct {
	OrderNumber      int            `json:"ordernumber"`
	City             string         `json:"city"`
	Zipcode          string         `json:"zipcode"`
	Address          string         `json:"address"`
	Phone1           string         `json:"phone1"`
	Phone2           string         `json:"phone2,omitempty"`
	Cancelled        bool           `json:"cancelled"`
	Delivered        bool           `json:"delivered"`
	DeliveryAttempts int            `json:"deliveryattempts"`
	Driver           DeliveryDriver `json:"deliverydriver"`
}

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{
	&Delivery{
		OrderNumber:      1,
		City:             "Here",
		Zipcode:          "52011",
		Address:          "Home",
		Phone1:           "6945123789",
		Phone2:           "2313722903",
		Cancelled: false,
		Delivered:        false,
		DeliveryAttempts: 0,
		Driver: DeliveryDriver{
			FirstName: "Mhtsos",
			LastName:  "Iwannou",
		},
	},
	&Delivery{
		OrderNumber:      2,
		City:             "There",
		Zipcode:          "1701",
		Address:          "Office",
		Phone1:           "6932728091",
		Cancelled: false,
		Delivered:        true,
		DeliveryAttempts: 1,
		Driver: DeliveryDriver{
			FirstName: "Lucas",
			LastName:  "Johnson",
		},
	},
	&Delivery{
		OrderNumber:      3,
		City:             "FarAway",
		Zipcode:          "920639",
		Address:          "Island",
		Phone1:           "6900777123",
		Cancelled: true,
		Delivered:        false,
		DeliveryAttempts: 24,
		Driver: DeliveryDriver{
			FirstName: "Pilotos",
			LastName:  "Aeroplanou",
		},
	},
}

// Constant for Bad Request
const BadReq string = `{"error": "bad request"}`
// Constant for Not Found
const NotFound string = `{"error": "not found"}`

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(deliveries)
}

// Read a specific delivery
func GetDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

// Create a new delivery
func AddDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	d := &Delivery{}
	// because deliveries start from 1
	orderNum := int(len(deliveries)) + 1
	err := json.NewDecoder(r.Body).Decode(d)
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	// Override whatever the sender had sent
	d.OrderNumber = orderNum
	// Append the delivery to deliveries list
	deliveries = append(deliveries, d)
	w.WriteHeader(http.StatusCreated)
	// Send back the delivery that was saved
	json.NewEncoder(w).Encode(d)
	return
}

func main() {
	// Set up router
	router := mux.NewRouter()
	// Set up subrouter for api version 1
	apiV1 := router.PathPrefix("/api/v1").Subrouter()
	// Set up routes
	apiV1.HandleFunc("/deliveries", GetAllDeliveries).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery/{ordernumber}", GetDelivery).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery", AddDelivery).Methods(http.MethodPost)
	// Start http server
	log.Fatal(http.ListenAndServe(":8000", router))
}

We’ve also added another route that is used when the HTTP method is POST which calls the AddDelivery function we just created. In order to test it out, I made this delivery.json file that you can also use. This is what it contains:

{
  "city": "ExperimentValley",
  "zipcode": "0000",
  "address": "REDACTED",
  "phone1": "6900999111",
  "cancelled": false,
  "delivered": false,
  "deliveryattempts": 3,
  "deliverydriver": {
    "firstname": "Mike",
    "lastname": "REDACTED"
  }
}

I saved it in my home directory with the name delivery.json and from the same place, after starting the server with go run main.go I ran:

`1`	`curl -H "Content-Type: application/json" -X POST --data-binary '@delivery.json' http://localhost:8000/api/v1/delivery`

to send it over to the server which then returned

{"ordernumber":4,"city":"ExperimentValley","zipcode":"0000","address":"REDACTED","phone1":"6900999111","cancelled":false,"delivered":false,"deliveryattempts":3,"deliverydriver":{"firstname":"Mike","lastname":"REDACTED"}}

which as we can see has ordernumber set to 4 even though I didn’t explicitly set it. Feel free to try setting ordernumber to 27 or something inside delivery.json and see what you get.

Upd8 1

Next up we’re going to update an entry. The appropriate HTTP method for this is PUT and the status code is http.StatusAccepted or http.StatusOK and I prefer the first one so that’s what I’ll use. The UpdateDelivery function we’re going to make has elements we saw previously in GetDelivery as well as AddDelivery. Not much of this is new so let’s see how we get things done.

// main.go
package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
	"strconv"
)

type DeliveryDriver struct {
	FirstName string `json:"firstname"`
	LastName  string `json:"lastname"`
}

type Delivery struct {
	OrderNumber      int            `json:"ordernumber"`
	City             string         `json:"city"`
	Zipcode          string         `json:"zipcode"`
	Address          string         `json:"address"`
	Phone1           string         `json:"phone1"`
	Phone2           string         `json:"phone2,omitempty"`
	Cancelled        bool           `json:"cancelled"`
	Delivered        bool           `json:"delivered"`
	DeliveryAttempts int            `json:"deliveryattempts"`
	Driver           DeliveryDriver `json:"deliverydriver"`
}

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{
	&Delivery{
		OrderNumber:      1,
		City:             "Here",
		Zipcode:          "52011",
		Address:          "Home",
		Phone1:           "6945123789",
		Phone2:           "2313722903",
		Cancelled: false,
		Delivered:        false,
		DeliveryAttempts: 0,
		Driver: DeliveryDriver{
			FirstName: "Mhtsos",
			LastName:  "Iwannou",
		},
	},
	&Delivery{
		OrderNumber:      2,
		City:             "There",
		Zipcode:          "1701",
		Address:          "Office",
		Phone1:           "6932728091",
		Cancelled: false,
		Delivered:        true,
		DeliveryAttempts: 1,
		Driver: DeliveryDriver{
			FirstName: "Lucas",
			LastName:  "Johnson",
		},
	},
	&Delivery{
		OrderNumber:      3,
		City:             "FarAway",
		Zipcode:          "920639",
		Address:          "Island",
		Phone1:           "6900777123",
		Cancelled: true,
		Delivered:        false,
		DeliveryAttempts: 24,
		Driver: DeliveryDriver{
			FirstName: "Pilotos",
			LastName:  "Aeroplanou",
		},
	},
}

// Constant for Bad Request
const BadReq string = `{"error": "bad request"}`
// Constant for Not Found
const NotFound string = `{"error": "not found"}`

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(deliveries)
}

// Read a specific delivery
func GetDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

// Create a new delivery
func AddDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	d := &Delivery{}
	// because deliveries start from 1
	orderNum := int(len(deliveries)) + 1
	err := json.NewDecoder(r.Body).Decode(d)
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	// Override whatever the sender had sent
	d.OrderNumber = orderNum
	// Append the delivery to deliveries list
	deliveries = append(deliveries, d)
	w.WriteHeader(http.StatusCreated)
	// Send back the delivery that was saved
	json.NewEncoder(w).Encode(d)
	return
}

// Update an existing delivery
func UpdateDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			err := json.NewDecoder(r.Body).Decode(d)
			if err != nil {
				w.WriteHeader(http.StatusBadRequest)
				w.Write([]byte(BadReq))
				return
			}
			d.OrderNumber = orderNum
			w.WriteHeader(http.StatusAccepted)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

func main() {
	// Set up router
	router := mux.NewRouter()
	// Set up subrouter for api version 1
	apiV1 := router.PathPrefix("/api/v1").Subrouter()
	// Set up routes
	apiV1.HandleFunc("/deliveries", GetAllDeliveries).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery/{ordernumber}", GetDelivery).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery", AddDelivery).Methods(http.MethodPost)
	apiV1.HandleFunc("/delivery/{ordernumber}", UpdateDelivery).Methods(http.MethodPut)
	// Start http server
	log.Fatal(http.ListenAndServe(":8000", router))
}

As we can see, nothing special, same stuff we’ve done before. Check order number, return error bad request if it’s bad. Go through the deliveries to find the one, if it doesn’t exist return not found. If it’s found, try to unpack the sender’s json to the existing delivery and if that fails return bad request otherwise save changes and return the delivery that was saved in our list. Note that we also override the value that the sender gave to keep our deliveries the way we want them and not have duplicates of what is supposed to be a unique identifier. Also we added the route on the bottom to make it usable. Similar to what we did to test it earlier, start the server with go run main.go and then run the following to change the data of delivery number 2:

`1`	`curl -H "Content-Type: application/json" -X PUT --data-binary '@delivery.json' http://localhost:8000/api/v1/delivery/2`

which returns:

{"ordernumber":2,"city":"ExperimentValley","zipcode":"0000","address":"REDACTED","phone1":"6900999111","cancelled":false,"delivered":false,"deliveryattempts":3,"deliverydriver":{"firstname":"Mike","lastname":"REDACTED"}}

and we can tell from ordernumber that it saved it there. To make sure, we can also run

`1`	`curl http://localhost:8000/api/v1/deliveries`

to get a list of all deliveries.

Delete one

I know that Delete is left and we will implement it but we’re not actually going to delete anything. This is an important point to make, we have to consider what the application is doing. If we want to have a history of deliveries, we don’t want any of them to be lost. What we’re going to do instead is set the Cancelled to true. Let’s think about why for a bit. If the order is successfully delivered, we use the UpdateDelivery method to set it and it’s all good. So it’s the job of DeleteDelivery to handle deliveries that need to be invalidated in some other way, not completed deliveries. The only reason I can think of that you’d want to “delete” a delivery is if the order being cancelled so that’s what we’ll do. We do the normal check for the path parameter, if we find a delivery with that number we set it to cancelled and if not we return error not found. Here is the code:

// main.go
package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
	"strconv"
)

type DeliveryDriver struct {
	FirstName string `json:"firstname"`
	LastName  string `json:"lastname"`
}

type Delivery struct {
	OrderNumber      int            `json:"ordernumber"`
	City             string         `json:"city"`
	Zipcode          string         `json:"zipcode"`
	Address          string         `json:"address"`
	Phone1           string         `json:"phone1"`
	Phone2           string         `json:"phone2,omitempty"`
	Cancelled        bool           `json:"cancelled"`
	Delivered        bool           `json:"delivered"`
	DeliveryAttempts int            `json:"deliveryattempts"`
	Driver           DeliveryDriver `json:"deliverydriver"`
}

// type deliveries is slice of Delivery pointers
type deliveryList []*Delivery

// variable deliveryList is of type deliveries
var deliveries deliveryList = []*Delivery{
	&Delivery{
		OrderNumber:      1,
		City:             "Here",
		Zipcode:          "52011",
		Address:          "Home",
		Phone1:           "6945123789",
		Phone2:           "2313722903",
		Cancelled: false,
		Delivered:        false,
		DeliveryAttempts: 0,
		Driver: DeliveryDriver{
			FirstName: "Mhtsos",
			LastName:  "Iwannou",
		},
	},
	&Delivery{
		OrderNumber:      2,
		City:             "There",
		Zipcode:          "1701",
		Address:          "Office",
		Phone1:           "6932728091",
		Cancelled: false,
		Delivered:        true,
		DeliveryAttempts: 1,
		Driver: DeliveryDriver{
			FirstName: "Lucas",
			LastName:  "Johnson",
		},
	},
	&Delivery{
		OrderNumber:      3,
		City:             "FarAway",
		Zipcode:          "920639",
		Address:          "Island",
		Phone1:           "6900777123",
		Cancelled: true,
		Delivered:        false,
		DeliveryAttempts: 24,
		Driver: DeliveryDriver{
			FirstName: "Pilotos",
			LastName:  "Aeroplanou",
		},
	},
}

// Constant for Bad Request
const BadReq string = `{"error": "bad request"}`
// Constant for Not Found
const NotFound string = `{"error": "not found"}`

// Read all deliveries
func GetAllDeliveries(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(deliveries)
}

// Read a specific delivery
func GetDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

// Create a new delivery
func AddDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	d := &Delivery{}
	// because deliveries start from 1
	orderNum := int(len(deliveries)) + 1
	err := json.NewDecoder(r.Body).Decode(d)
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	// Override whatever the sender had sent
	d.OrderNumber = orderNum
	// Append the delivery to deliveries list
	deliveries = append(deliveries, d)
	w.WriteHeader(http.StatusCreated)
	// Send back the delivery that was saved
	json.NewEncoder(w).Encode(d)
	return
}

// Update an existing delivery
func UpdateDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			err := json.NewDecoder(r.Body).Decode(d)
			if err != nil {
				w.WriteHeader(http.StatusBadRequest)
				w.Write([]byte(BadReq))
				return
			}
			d.OrderNumber = orderNum
			w.WriteHeader(http.StatusAccepted)
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

// Delete a delivery (not really)
func DeleteDelivery(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	params := mux.Vars(r)
	orderNum, err := strconv.Atoi(params["ordernumber"])
	if err != nil {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte(BadReq))
		return
	}
	for _, d := range deliveries {
		if d.OrderNumber == orderNum {
			w.WriteHeader(http.StatusOK)
			d.Cancelled = true
			json.NewEncoder(w).Encode(d)
			return
		}
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(NotFound))
	return
}

func main() {
	// Set up router
	router := mux.NewRouter()
	// Set up subrouter for api version 1
	apiV1 := router.PathPrefix("/api/v1").Subrouter()
	// Set up routes
	apiV1.HandleFunc("/deliveries", GetAllDeliveries).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery/{ordernumber}", GetDelivery).Methods(http.MethodGet)
	apiV1.HandleFunc("/delivery", AddDelivery).Methods(http.MethodPost)
	apiV1.HandleFunc("/delivery/{ordernumber}", UpdateDelivery).Methods(http.MethodPut)
	apiV1.HandleFunc("/delivery/{ordernumber}", DeleteDelivery).Methods(http.MethodDelete)
	// Start http server
	log.Fatal(http.ListenAndServe(":8000", router))
}

First we start the server with go run main.go and then to test our newly added functionality, in my case with cURL, we can run:

`1`	`curl -X DELETE http://localhost:8000/api/v1/delivery/1`

and get back:

{"ordernumber":1,"city":"Here","zipcode":"52011","address":"Home","phone1":"6945123789","phone2":"2313722903","cancelled":true,"delivered":false,"deliveryattempts":0,"deliverydriver":{"firstname":"Mhtsos","lastname":"Iwannou"}}

Conclusion

That’s it for this part then. You now have a fully functional REST API written in Go that you can play around with. There are a few more things that you can do with gorilla/mux and the code can be made a little bit nicer so if you’re interested in that, check out my completed version. Thank you for reading and I hope you enjoyed.

REST Is Over

Sun, 04 Apr 2021 17:33:02 +0300

Web APIs are terrible

After watching a talk a few months back about REST vs GraphQL and the history of APIs, I agreed with the speaker that the API is supposed to be a way of communication, a language of sorts. In programming languages this codified way of communication is enforced by the compiler or the interpreter or the runtime. If you write invalid code in the programming language of your choosing, it won’t work. Something or other will complain or break and prevent you from miscommunicating. This got me wondering APIs especially web ones are so bad.

RESTing comfortably

With the introduction of what we call REST nowadays, basically JSON over HTTP, it’s all too common for application to use this mode of communication. How this is enforced in practice is very variable though. From different path schemes, to the same HTTP verbs used for different purposes, to (invariably bad) ways to communicate from machine to machine what the API is, it’s all up in the air. Most fairly complex things that have a REST API, don’t even expect you to use them by directly interacting with it but instead use client libraries. This has been going on for about a decade or two and it seems things are changing. Some thought that GraphQL would be the end of that and it’s a pretty good step up but adoption is nowhere near as high. In general REST is the assumed default way to interact with something programmatically. Should it be though? Is GraphQL the best replacement we’ve got? I don’t think so.

gRPCee what I did there

A somewhat recent thing called gRPC started picking up steam. Talks about it, articles about it, large software projects using it but not really prominent on the web. With this nifty little thing after writing a file that describes what your program does, you can generate some ready to use code, write the logic and grpc.Serve() to the moon. Even better, anyone wanting to write a client in any of the many supported languages can generate the boilerplate for a client from the same definition file and start calling the functions of the server really easily. I’m being a bit simplistic for the sake of brevity but it’s really cool, trust me. It’s also really fast. Alright, it’s better than freshly baked bread apparently so why is it not used on the web? Cause it’s only HTTP/2 and the support for that is pretty limited. Fear not though, we’re going to circle back to our crusty old frenemy, REST.

Support for REST is widespread and there is a webdev in every city block ready to POST some JSON to your endpoint. So we can’t immediately get rid of the kludgy wart but what about giving pixel pushers some REST while service to service communication is improved? You see, since definitions exist for what the service sends and receives, the types of all the things being passed around, it’s quite easy to use some simple annotations to auto-generate a REST API. This doesn’t 100% fix the problems that exist but it does smooth over some parts of it. Okay so that’s it? Not just that, since the API is standardized we can even auto-generate documentation for it! OpenAPI (formerly Swagger) docs can be easily generated and served to potential downstream users of the API. Brilliant I think.

Conclusions

Over the past few months I’ve been reading a lot about gRPC, grpc-gateway, different ideas and implementations, trying out quite a few of them but I’ve yet to see one that makes me super excited and go “Aha! this is The One” or whatever. I could do a link dump of about 100-150 tabs that I have open but that would be tiring and essentially do nothing to help. Currently I’ve been trying to decide among 2-3 options for how a service should be structured and such but I’m still testing out prototypes, haven’t reached any conclusions. However, I hope this was a semi-educational read and got you a bit interested in the ecosystem. That’s all for now, I hope you enjoyed reading.

pardon me

~~check out the tutorials section for my Go REST API Part 2 coming soon~~ sorry, I had to :^)

Go API Part 1

Thu, 01 Apr 2021 06:39:47 +0300

Intro

This is going to be a simple tutorial about creating a backend web service that can be accessed over a REST API. The application is going to be able to do something very basic just to make everyone familiar with the basics. There will eventually be a part 2 where we will make an application that performs the basic CRUD(Create Read Update Delete) actions, it will include real-life applicable data and actions as well as a database for storing information. Let’s start then.

Getting set up

I’m not going to cover how to set up the development environment but you can look at these links:

Greeting the world

First off, in a file called main.go we declare the package (important note here, if the package is not main then there is no ability to use go run on it), add a couple basic imports, write the main function and make it print something:

package main

import (
	"fmt"
)

func main() {
	fmt.Println("ordering facilities online")
}

And that’s our hello world right there. Use go run main.go to run the program.

Webbing it up

Alright now that that’s done, let’s start messing with the web capabilities. First we’ll select a port depending on if the PORT environment variable has been set or not and then we’ll write a simple endpoint that returns the current year. While we’re at it I’ll switch out fmt for log because I like the output better:

package main

import (
	"log"
	"net/http"
	"os"
	"strconv"
	"time"
)

func getCurrentYear(w http.ResponseWriter, r *http.Request) {
	currentTime := time.Now()
	year := currentTime.Year()
	yearString := strconv.Itoa(year)
	w.Header().Set("Content-Type", "text/plain")
	w.WriteHeader(http.StatusOK)
	w.Write([]byte("current year is: "+yearString))
	return
}

func main() {
	log.Println("ordering facilities online")
	port := os.Getenv("PORT")
	if port == "" {
		port = "8080"
	}
	http.HandleFunc("/year", getCurrentYear)
	log.Fatal(http.ListenAndServe(":"+port, nil))
}

A couple things to comment on. First off, := means that you don’t have to manually declare the type of the variable, it will be figured out automatically by the compiler. Second, the signature of the getCurrentYear function. The arguments might look weird: w http.ResponseWriter, r *http.Request but those two variables are passed to the function by http.HandleFunc. ResponseWriter w allows us to work with what we’ll return and Request r gives us info about the request. Let’s get into the getCurrentYear function. The first thing it does is get the year which is of type int and then using string converter (strconv) to make it a string in order to be able to be added together with our message in the end. After creating yearString we se the content type so the client (in this case the browser) knows what we’re sending and set the status to OK (code 200) so the client knows how things went. The responsewriter writes a slice (indicated by []) of bytes to the client which in this case is the message current year is: along with the year as calculated previously. Inside the main function we have http.HandleFunc("/year", getCurrentYear) which says that if you visit /year then the function getCurrentYear should be run. This function is also called a handler because it handles the request. Lastly, we run http.ListenAndServe(":"+port, nil) inside log.Fatal because it returns an error if it exists so if any error with starting the http server happens, we’ll see it and the program won’t run. To test all of this, run go run main.go and then open a browser, visit localhost:8080/year and see the magic.

JSONify our Year

To give a simple introduction to data structures and the JSON format we’ll work with. First we’ll make a struct which will be our custom type for representing the year with some annotations. We’ll also import encoding/json to translate the struct to json based on the annotations and return it to the client. In addition, we’ll adjust some code accordingly.

package main

import (
	"encoding/json"
	"log"
	"net/http"
	"os"
	"time"
)

type MyYear struct {
	CurrentYear int    `json:"currentyear"`
	Good        bool   `json:"good,omitempty"`
	Comment     string `json:"comment,omitempty"`
}

func getCurrentYear(w http.ResponseWriter, r *http.Request) {
	currentTime := time.Now()
	year := currentTime.Year()
	var goodYear bool
	var comment string
	if (year == 2020 || year == 2021) {
		goodYear = false
		comment = "big oof"
	}
	myCurrentYear := MyYear{
		CurrentYear: year,
		Good: goodYear,
		Comment: comment,
	}
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(myCurrentYear)
	return
}

func main() {
	log.Println("ordering facilities online")
	port := os.Getenv("PORT")
	if port == "" {
		port = "8080"
	}
	http.HandleFunc("/year", getCurrentYear)
	log.Fatal(http.ListenAndServe(":"+port, nil))
}

Alright so there is a lot to go over. First off, the struct MyYear is our custom type. It has 3 fields, CurrentYear to store the year as an integer, Good to store a boolean about the year being good or not and Comment for the year. The things next to each field in the struct specify the name that the json field will have as well as if it should be printed when the field is empty (specified by omitempty). The content type was changed to application/json since that is what we’re returning. In the code above, myCurrentYear is created when the request is sent according to some conditions. There is a little if statement that sets a default value for Good and Comment if the year is 2020 or 2021. The variables goodYear and comment are declared without a default value and if we hadn’t used them the compiler would error out but we did so that’s not an issue just something to keep in mind. All of the translation of the struct to json is done automatically, we just created a new json encoder that encodes into the responsewriter aka to the client and then encoded our struct. If you run go run main.go now and then visit localhost:8080/year you will see that now the response is in json, exactly what we wanted.

Adding a dependency

Eventually we will want to be able to do a GET request on something like example.com/api/v1/orders/521 and get back the details of the 521st order in JSON format. This requires some routing as well as reading path parameters, in this case that parameter is 521. For this purpose we’ll use a request multiplexer, specifically gorilla/mux. To add it to our system we can just do go get github.com/gorilla/mux and then add it as a dependency as shown below:

package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
	"os"
	"time"
)

type MyYear struct {
	CurrentYear int    `json:"currentyear"`
	Good        bool   `json:"good,omitempty"`
	Comment     string `json:"comment,omitempty"`
}

func getCurrentYear(w http.ResponseWriter, r *http.Request) {
	currentTime := time.Now()
	year := currentTime.Year()
	var goodYear bool
	var comment string
	if (year == 2020 || year == 2021) {
		goodYear = false
		comment = "big oof"
	}
	myCurrentYear := MyYear{
		CurrentYear: year,
		Good: goodYear,
		Comment: comment,
	}
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(myCurrentYear)
	return
}

func main() {
	log.Println("ordering facilities online")
	port := os.Getenv("PORT")
	if port == "" {
		port = "8080"
	}
	router := mux.NewRouter()
	router.HandleFunc("/year", getCurrentYear)
	log.Fatal(http.ListenAndServe(":"+port, router))
}

We’ve added it, notice how http.ListenAndServe has a second argument now instead of nil. Also instead of http.HandleFunc we have router.HandleFunc since the handler is no longer the default one from net/http. Now if you visit localhost:8080/year you’ll notice that json is returned just like before. I didn’t comment on it previously but did you notice the lack of good in the json response? Why is that? We explicitly set a value so what’s the deal there? In the case of a bool variable, false is considered to be the “empty” state of the variable so it’s not shown. All we need to do to fix it is remove the omitempty from its annotation (as well as the comma) and we’ll get the result we want. Here is the finished code then:

package main

import (
	"encoding/json"
	"github.com/gorilla/mux"
	"log"
	"net/http"
	"os"
	"time"
)

type MyYear struct {
	CurrentYear int    `json:"currentyear"`
	Good        bool   `json:"good"`
	Comment     string `json:"comment,omitempty"`
}

func getCurrentYear(w http.ResponseWriter, r *http.Request) {
	currentTime := time.Now()
	year := currentTime.Year()
	var goodYear bool
	var comment string
	if (year == 2020 || year == 2021) {
		goodYear = false
		comment = "big oof"
	}
	myCurrentYear := MyYear{
		CurrentYear: year,
		Good: goodYear,
		Comment: comment,
	}
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(myCurrentYear)
	return
}

func main() {
	log.Println("ordering facilities online")
	port := os.Getenv("PORT")
	if port == "" {
		port = "8080"
	}
	router := mux.NewRouter()
	router.HandleFunc("/year", getCurrentYear)
	log.Fatal(http.ListenAndServe(":"+port, router))
}

Adding gorilla/mux didn’t do a lot, we have added no extra functionality to our program but added a dependency which is generally not good. The next part will focus more on the awesome things we can do with gorilla/mux related to paths, subrouters, regular expressions and matching HTTP verbs.

Conclusion

This was a pretty heavy introduction to Go with an emphasis on using it for web services. Feel free to run the examples, tweak them, add to them and become familiar with documentation. Next time we’ll be getting into full CRUD operations, databases and probably a Dockerfile too. Thank you for reading, I hope you enjoyed and maybe learned something.

Golang Is Fucking Rad

Thu, 18 Mar 2021 21:21:15 +0200

Radical Gopherism

I’ve been using Go for the better part of a year at this point and I’m loving most things about it. You know the typical stuff, types, compiled, fast, portable, easy to cross-compile and the rest of the goodness. However that’s not what made me stick with it although those things helped. The main things I liked were mostly that it was in many ways not like Java. There is no need for an IDE, the language is not endlessly complex, I can read other people’s code right away. Don’t forget the compiler-enforced formatting, easy static compilation allowing for tiny container images. The standard library is also phenomenal if frameworks are not your cup of tea it’s very possible to avoid them in a lot of cases. And the new file embedding functionality? Amazing, my site is now a portable self-contained binary. And the adoption? Practically any project I’m interested in just so happens to be written in it! Most things I use, kubernetes, docker, terraform, vault, gitea, droneci, fluxcd, minio and more I’m probably forgetting.

Reality

Time to stop drooling though. While I wasn’t inaccurate or misrepresentative above, I want to look at pros-cons for a moment.

Dank Reality

The main thing I do is write backend web microservices and small operating system utilities. As mentioned previously, the stdlib is amazing and quite a few things I’ve written use that almost exclusively. If web frameworks are needed, there are plenty from piece-by-piece like gorilla, go-kit(depends if you count it) or go-chi, to all-encompassing gargantuan ones like go-micro (ironic name, I know) to middleground solutions like gin-gonic. I can find client libraries for everything I want be it any sort of api, databases, application messaging systems and recently I found out I can even run ML models generated by keras in python from go. I didn’t harp too much on this before but the static compilation is sweet. Using docker multi-stage builds and starting from the 150MB fat golang image with devtools in it but then being able to run the resulting binary in a run stage based on scratch and have a 3MB production image is incredible. I use it more and more every day and basically haven’t looked to shell scripting for anything except heavy string manipulation but even there I’m starting to move as see in gopasswd.

Sad Reality

This site is my doomer cabin in the woods so the fun stops here. Also lists are scary so let’s go

language package manager: having maintained gentoo ebuilds, I’ll never not hate language package managers
not very competitive alternative compiler: gcc-go produces slower artifacts and I don’t want a dotnet situation
google is very heavily involved: the pkg.go.dev shit just showcased it but I’m not comfortable with google being involved and yes this goes for k8s as well
grpc support: google came up with grpc and is too involved in go but you need a 3rd party plugin for protoc to output go which is stupid
dev2prod: translating from local to remote dependencies and vice versa is handled poorly
modules: kinda weird, I’ve come across too many cases where rm go.mod go.sum && go mod init && go mod tidy was needed
generics: don’t want generics but everyone is begging and the advantage of small and understandable might be lost in the future

Conclusion

Good language, fits what I do, like using it, will continue using it, wish some stuff was improved or fixed. That was all for this post, I hope you enjoyed.

Programming on inherent site

One Billion Rows Challenge

The challenge

My proposed alternative challenge

Solution in Go

The TempStat struct

Opening the file

Processing the input

Round and Print

Bringing it together in main()

Benchmarks

How to get started

Conclusion

How This Website Is Made

Intro

Beginning

Evaluation

Improvements

Solutions

Hugo

Transitioning

Directory Structure

Layouts

CSS

Docker

Impressions

Conclusion

Go API Part 3

Previous relevant tutorial

Intro

Starting point

Restructuring

Important note

Models

API

Main after API restructure

List Restructure

Main after list database restructure

Database

Interface

Separating concerns

List database

Refactor API code

Postgres

Connection code

Choose database

Test connection

Queries

Create table query

See all deliveries

See one delivery

Add one delivery

Update one delivery

Delete one delivery

Run last test

Conclusion

The Power of the Library Example

The Library example

Data Structures

Functions

Interfacing

Conclusion

Go API Part 2

Previous relevant tutorial

Intro

Starting point

Get all

Get one

Add one

Upd8 1

Delete one

Conclusion

REST Is Over

Web APIs are terrible

RESTing comfortably

gRPCee what I did there

Conclusions

pardon me

Go API Part 1

Intro

The `TempStat` struct

Bringing it together in `main()`