After reading one book, taking a class, and a couple false starts, Kubernetes was very much in front of me as something I needed to learn. As a long time home-lab herder, often a great way for me to learn is to get hands-on. So I decided to build a new lab and start migrating some services to it.

With just a little bit of research and talking to some knowledgeable folks, I landed on using K3s on a three node cluster on fan-less hardware backed with a PostgreSQL database somewhere off-cluster.

Once I had the base operating systems running and remote access configured, the K3s install was pretty straight forward. There was a little back and forth in some GitHub issues about getting IPv6 working, but the community was very helpful and ultimately, the one liners below got me going.

export TERM=xterm
export DATASTORE_ENDPOINT="postgres://k3s:$DB_PASSWORD@db4.l.larch.space:5432/k3s"

curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=v1.22 sh -s \
- server \
--datastore-endpoint="$DATASTORE_ENDPOINT" \
--node-ip="172.16.15.20,fc15::20" \
--cluster-cidr="10.42.0.0/16,fc15:1::/56" \
--service-cidr="10.43.0.0/16,fc15:2::/112" \
--disable-network-policy \
--tls-san=k.znet \
--cluster-domain=k.znet

My hope is that by setting cluster-domain, I can stub the DNS zone and implement my existing DNS.

Get the token from the first server in /var/lib/rancher/k3s/server/node-token and export it.

export TOKEN="nope"

Now install the remaining nodes, using unique IPs assigned to each node’s interface.

curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=v1.22 K3S_TOKEN=$TOKEN sh -s \
- server \
--datastore-endpoint=$DATASTORE_ENDPOINT \
--node-ip="172.16.15.21,fc15::21" \
--cluster-cidr="10.42.0.0/16,fc15:1::/56" \
--service-cidr="10.43.0.0/16,fc15:2::/112" \
--disable-network-policy \
--tls-san=k.znet \
--cluster-domain=k.znet

curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=v1.22 K3S_TOKEN=$TOKEN sh -s \
- server \
--datastore-endpoint=$DATASTORE_ENDPOINT \
--node-ip="172.16.15.22,fc15::22" \
--cluster-cidr="10.42.0.0/16,fc15:1::/56" \
--service-cidr="10.43.0.0/16,fc15:2::/112" \
--disable-network-policy \
--tls-san=k.znet \
--cluster-domain=k.znet

At first I was clever, but then I submitted to the will of whoever it was that thought I wanted a /56 or a /112, but there are hard-codes or some such.

Happy New Year!