This blog is to document a solution for avoiding the ongoing costs of learning about kubernetes clustering on a cloud platform. Its a bit opinionated.

Most of us who show an interest in kubernetes are limited in the access we can gain to an environment in which we can learn, without having to hand over credit card details.

Initially we get offered credits on this or that cloud based environment as an inducement to start a path of dependence on a platform that requires an ongoing commitment of financial payment to maintain access to data/content held captive by the platform.

I took advantage of one, and before I knew it my time and credit was up. I didn't get very far. Thankfully, I didn't upload anything I wanted to keep.

There is a growing problem that too much content is ending up on platforms that can hold your data hostage unless a payment is made. Access to that data is subject to many conditions. The corporatisation and centralisation of computing is eroding the freedom that the internet promised. We need to find other ways.

So I started to look into single board computers (sbc) as an alternative and found a number of blogs that provided guidance on how to build a kubernetes cluster on raspberry pi's. The docker pirate site provided hypriot and other bits.

Some of the significant advantages were cost - the cluster had 4 pi's, a multiport usb charger to power it and a 5 port switch - all up probably about A$400. Ongoing costs of power are much better than a standard pc with a 400 watt power supply. Another advantage is desktop geography - it takes up very little space.

An early photo of the pi cluster

And its still going - I've deployed ghost and wordpress and learnt a lot. Not least was how to deploy metallb as a load balancer and traefik as a containerised proxy.

But - the architecture was armv7 and many common solutions aren't supported (Helm support for armv7 is not good). It takes a bit of hacking to get some stuff to work. All good fun, but the pi's only have 1 gb ram. (newer pi's can have 8gb - so I'm interested again)

So I started to look for a cheap, low power x86 platform.

There's some low cost chinese systems around, there's asus tinkerboard and lattepanda and barebones boxes, but the costs starts to escalate.

I looked at auction sites for second hand stuff and noticed an add for a Dell Wyse Thin client and then went on a google hunt and found https://www.parkytowers.me.uk/thin/. This is a great site that provides fabulous information on re-purposing thin clients.

The thin clients I found were Dell Wyse Z90D7 models - Dual core AMD G-T56N
1.65GHz, 4 GB DDR3 RAM (max 16 Gb), 4 x USB2.0, 2 x USB3.0, and 16 GB sata DOM (disk on module).

I got 4 of them for A$100.

I had 4 x 8GB, 24 pin DDR3 1333 ram DIMMs that I salvaged from a previous workstation, and so ended up with a 16 Gb server and 3 8 Gb nodes. I bought a 240 GB sata ssd and 3 120 Gb sata ssds and 4 cables that enabled all the disk space I needed. It just worked. The ssds were small enough to be packed into the case without too much trouble.

A look inside

and with the ssd.

Part of my motivation was to gain experience of some of the technologies emerging at work. Terraform was being used in one new project, along with Openshift. Openshift is based on kubernetes and docker, but its a Redhat packaged solution, commercial and expensive, plus it shields the user from some of the underlying configuration issues. Terraform, however, is an open source, infrastructure as code tool. It does Openshift, but there are also recipes for AWS, Kubernetes, Azure amongst others including Vsphere. What I was looking for was a way of learning the use of infrastructure as code, as well as container clustering and experiment with the idea of immutable systems maintenance - deploying a replacement system, rather than patching the old one. This required a cluster node os, designed for the update replacement model (immutability).

At first I looked at container linux, coreos, but their recent purchase by Redhat meant that their energy was directed into integration with a new corporate entity rather than maintenance of the documentation. I attempted to follow their docs and this started me along the path of installing Matchbox, but I couldn't get their Terraform examples to work.

Basically, Matchbox is a service, that will enable iPXE booting a system, and deployment of an OS, controlled by configuration files, that allow specification of a particular version (that can be pinned, and increased).

The Coreos Terraform kubernetes example used a Typhoon module to deploy kubernetes, but it was not working. However as I attempted to debug, I had a look at the Typhoon site - which was more up to date and reviewed and amended the configuration. (note the site has been updated and no longer supports Container Linux, but does support Flatcar Linux) And it started to work.

Wow - 3 node kubernetes cluster on bare metal, deployed with Terraform.

And there's more, Typhoon have another bare metal example that deploys Fedora Atomic. Fedora Atomic is interesting for two reasons - 1. Its been designed as an immutable platform with atomic ostree updating and 2. Coreos has been purchased by Redhat, who have a downstream version of Fedora Atomic - the announced intention is to merge Coreos Container Linux with Fedora Atomic.