Use Kairos to run a Kubernetes cluster

kubernetes home-infra selfhosting home-lab k0s kairos
Use Kairos to run a Kubernetes cluster

I use Kubernetes in my day job, and as a freelance DevOps/cloud/SRE engineer it’s close to a requirement these days — it’s become the de facto standard for running container workloads. So when I built out my home-infra, running a Kubernetes cluster was part of the plan from day one.

Why not Talos

Talos is very popular, and for good reason — I run it myself on Hetzner, and it’s part of a service we’re building at Autops. But I like exploring alternatives, and I’m not fully “all-in” on Talos for two reasons:

  • License risk at the ecosystem level. A bunch of open-source projects I like (Terraform, Docker Desktop) have shifted to more restrictive commercial licenses over the past few years, leaving communities to fork or scramble. To be fair, Talos Linux itself is MPL-2.0 and stays fully open and usable in production — it’s Sidero Labs’ management layer, Omni, that’s under a Business Source License (BUSL) for production use. Still, that split between “free OS, paid management plane” is a pattern I want to be careful about depending on too heavily. But I do get that as a company you need to generate revenue in some way.
  • A different management model. With Talos you don’t manage the OS in the traditional sense — you talk to an API to configure it, troubleshoot, and do day-2 operations. That’s genuinely nice for automation, but it also means no shell, no ssh, and no familiar Linux toolbox to fall back on when something goes sideways.

Kairos

I recently discovered Kairos, and even though it’s still a young project, it caught my interest because it lets you build your own OS images using bootc. I already use bootc for my own Linux desktop, so this felt like a natural fit. Kairos was also accepted into the CNCF Sandbox in April 2024, which makes it a bit less exposed to the kind of licensing surprises I mentioned above.

K0s

Kairos itself is not a Linux distribution — it’s a framework for building, configuring, and maintaining an immutable OS. Unlike Talos, which is purpose built to run Kubernetes and nothing else, Kairos doesn’t ship with Kubernetes at all.

Instead, Kairos lets you install a Kubernetes distribution on top using providers. K3s and K0s are supported natively, and a few community-maintained providers exist as well.

I’ve used K0s before and I did like it — it ships as a single, zero-dependency binary that bundles every control-plane component, which keeps the architecture refreshingly simple.

At one point I assumed the project would fade out with the aquisition by IREN, but it’s still actively maintained (it’s a CNCF-certified distribution, currently backed by Mirantis).

Configuration

The setup has two parts: a custom OS image that boots and installs itself unattended, and a tiny discovery service that hands each machine its own config based on who it is.

Building the image

Kairos images are built like a container image. I start from Kairos' Hadron base and use kairos-init to layer in the K0s provider:

# https://quay.io/repository/kairos/kairos-init?tab=tags&tag=latest
FROM quay.io/kairos/kairos-init:v0.17.3 AS kairos-init
FROM ghcr.io/kairos-io/hadron-toolchain:v0.0.1-beta1 AS bottom
FROM ghcr.io/kairos-io/hadron:v0.5.1 AS base
ARG VERSION

RUN --mount=type=bind,from=kairos-init,src=/kairos-init,dst=/kairos-init \
  /kairos-init -l debug -s install --version "${VERSION}" --provider k0s --provider-k0s-version v1.36.4+k0s.0 && \
  /kairos-init -l debug -s init --version "${VERSION}"

podman build --build-arg=VERSION=0.1.0 -t registry.example.com/mystuff/kairos:0.1.0 -f Containerfile.k0s .

That image gets pushed to my registry, then turned into a bootable ISO with auroraboot:

The reason I needed to push to a remote registry is that auroraboot doesn’t seem to work with local images. At least I could not get it working.

podman run -v $PWD/cloud-config.yaml:/cloud-config.yaml \
  -v "$XDG_RUNTIME_DIR/containers/auth.json:/run/auth/config.json:ro,z" \
  -e DOCKER_CONFIG=/run/auth \
  -v $PWD/iso-out:/tmp/auroraboot \
  --rm -ti quay.io/kairos/auroraboot:v0.27.0 \
  --set container_image=oci:registry.example.com/mystuff/kairos:0.1.0 \
  --set "disable_http_server=true" \
  --set "disable_netboot=true" \
  --cloud-config /cloud-config.yaml \
  --set "state_dir=/tmp/auroraboot"

One image, many nodes

I wanted to burn a single ISO and use it on every node — controllers and workers alike — rather than build a separate image per machine. Kairos makes this easy through the cloud-config’s templating: instead of baking in per-node settings, the config baked into the image only points to a discovery endpoint, keyed by the machine’s hardware uuid (read at boot via sysinfo):

#cloud-config
config_url: "http://10.0.0.10:8088/register?uuid={{ required \"uuid is required for discovery\" .Values.product.uuid }}"

Behind that URL sits a small Caddy server. It maps the uuid query parameter straight to a per-node .yaml file on disk:

@config_request path /register
handle @config_request {
	@uuid_param query uuid=*
	handle @uuid_param {
		rewrite * /{query.uuid}.yaml
		file_server
	}
	handle {
		respond "No configuration found for the specified UUID" 404
	}
}

So the flow at boot is:

  1. Node boots the generic ISO
  2. Asks Caddy “who amI?”
  3. Caddy serves back that node’s real config
  4. Kairos applies it and installs itself, unattended.

The per-node config

Each node file sets the hostname, a static network config, and whether the node is a K0s controller or worker. The controller looks like this:

#cloud-config
stages:
  initramfs:
    - name: "Setup hostname"
      hostname: ctrl-01.c01.home.infra.example.internal
    - name: "Setup network files"
      files:
        - path: /etc/systemd/network/10-eno1.network
          content: |
            [Match]
            Name=eno1
            [Network]
            Address=10.0.1.10/24
            Gateway=10.0.1.1
            DNS=10.0.1.53

k0s:
  enabled: true

install:
  device: "/dev/sda"
  reboot: true
  auto: true

Workers look almost the same, but flip k0s for k0s-worker and point at a join token (generated on the controller with k0s token create --role=worker):

k0s-worker:
  enabled: true
  args:
    - --token-file /etc/k0s/token

write_files:
  - path: /etc/k0s/token
    permissions: "0600"
    content: |
      H4sIAAAAAAAC/2xV0Y...

Workers also get an extra, unformatted partition reserved for storage — handy later for something like Longhorn or Rook:

install:
  extra-partitions:
    - name: k8s-storage
      size: 409600
      fs: none

With that in place, adding a node is just: plug in the ISO, boot, and let Kairos figure out the rest from its uuid.

In the following posts I’ll cover how day-2 operation the cluster look like. I still need to figure that out but came across some intereating option in the documentation