Reduce the Pain

I started with the idea of learning about kubernetes. And I've done a bit of that, at least to the depth I'm prepared to dive. Its actually a mandelbrot set. The complexity never ends.
But unless you do it for a living, its probably worth considering time frames.
It should have some point, so I'm starting to think about driving the car, rather than letting it drive me.

Choosing what to do

I'm not a coder, more a sys admin, which means I implement, but it also means I like to have finished product, and appreciate it working, doing what its supposed to do - to realise purpose.
I reflected that I was being driven by product development, and it was frustrating me. You can't run a railroad that way, you have to accept a certain level of constraint based on the technology that is in place. There might be a new model locomotive available from the vendor but the one you have, hasn't worn out, and its the same guage as the track.
So I've decided to build the platform and then spend time, using it. And not be drawn into updating it until I'm ready.

Experience

This time was going to be different. I thought the upgrade would be fairly straight forward, but it wasn't. The current cluster has Coreos Container Linux as the OS . Poseidon Typhoon is the kubernetes distribution. Its nicely engineered, clean and sophisticated, smart, open source. It uses Terraform and Matchbox to build the cluster. Its a bit like magic.
I was running a little over specced for the workload. So I pulled one node out and repurposed the matchbox/nfs server I used for the initial build which give me two nodes, the initial cluster is happy with only two nodes as well. I could keep going and when ready - cut across the applications.
But the next iteration, with Fedora Coreos didn't fly for me. Fedora Coreos is Redhat's mutation of Coreos Container Linux. They've raced to get it onto the market after Redhat vacuumed up Coreos and it broke on my hardware. Lots of new stuff, but its not quite stable yet, at least that seems to be the case on my old, under powered hardware.
However, Typhoon also has a variant that uses Flatcar Container Linux. but that didn't work either.
But the trail was warm.
The people who do Flatcar Linux have also developed an open source kubernetes distribution called Lokomotive, and after some iterations, its working. (There is a bug, but its been identified and should be fixed in the next release - soon I hope)
Its not for the faint hearted. Setting up an IPXE boot environment (using dnsmasq) to deploy the OS was time consuming, but I had already done that for Typhoon.
Its going to stay for a while.