Building a High Availability Kubernetes Cluster Across Mixed Hardware (Part 1: The Build)

What Started This

My Kubernetes cluster had a problem I'd been ignoring for months: one control plane node. A single Proxmox VM running etcd, the API server, the scheduler, all of it. If that VM went down, even for a routine Proxmox update, the entire cluster was unreachable. No kubectl, no deployments, no ArgoCD syncs. Every service that runs on K8s (Vaultwarden, MarketMind, ArgoCD) just... stops.

And this wasn't a matter of if. It was when. My Proxmox server had a faulty RAM stick that would cause random restarts. No warning, just gone. Every time it happened, my notes app and password manager disappeared with it. Those two are the worst services to lose unexpectedly. When you need a password, you need it right now. When you need your notes, you need them right now. Not in five minutes when the VM finishes booting back up.

I already built an HA load balancer with keepalived and Caddy across three physical machines, specifically so that no single failure could take down my service routing. But the irony was thick: the LB was highly available, and the thing it was routing traffic to was not.

I had two BeeLink mini PCs sitting on my desk, freshly set up with Ubuntu and wired into the network. I'd been telling myself they were "for the HA cluster" for weeks. Time to actually do it.

The Plan

1. Provision a new VM on Proxmox via Terraform (done)
2. Install K8s prerequisites on all 3 nodes via Ansible (done)
3. Initialize the first control plane with kube-vip for API server HA (done)
4. Join the BeeLinks as additional control plane nodes (done)
5. Untaint all nodes so they also run workloads (done)
6. Install Traefik, migrate workloads from old cluster (Part 2)
7. Retarget HA LB and decommission old cluster (Part 2)

The Hardware

This is where it gets fun. Most HA K8s guides assume you have three identical machines. I don't. My cluster runs across a VM and two bare metal mini PCs, spread across different physical machines for real fault tolerance.

The Proxmox VM lives on a Supermicro server. The BeeLinks are standalone boxes plugged into the same switch. If Proxmox goes down, n1 and n2 keep the cluster alive. If one BeeLink dies, the other two nodes maintain quorum. That's the whole point.

All three nodes run as both control plane and worker. With only three machines, dedicating any of them to just control plane duties would waste too much compute. And kubeadm makes it easy to retaint them later when more hardware shows up.

Step 1: Provision the Proxmox VM

I already have Terraform managing my Proxmox VMs (the old cluster, the GitHub runner, the WireGuard server). Adding a new one is just another .tf file.

terraform/proxmox/ha-cluster.tf:

resource "proxmox_virtual_environment_vm" "ha_cp" {
  name      = "ha-cp"
  node_name = var.proxmox_node
  vm_id     = 140

  clone {
    vm_id = var.template_vm_id  # Ubuntu 24.04 cloud-init template
  }

  agent { enabled = true }

  cpu {
    cores = 4
    type  = "x86-64-v2-AES"
  }

  memory { dedicated = 16384 }

  disk {
    interface    = "scsi0"
    size         = 150
    datastore_id = var.datastore_id
  }

  initialization {
    ip_config {
      ipv4 {
        address = "192.168.1.42/24"
        gateway = var.gateway
      }
    }
    dns { servers = var.dns_servers }
    user_account {
      keys     = [trimspace(file(var.ssh_public_key_path))]
      username = "ubuntu"
    }
  }

  network_device {
    bridge   = var.bridge
    firewall = true
  }

  operating_system { type = "l26" }
  scsi_hardware = "virtio-scsi-single"
  on_boot       = true
}

cd terraform/proxmox
terraform plan -target=proxmox_virtual_environment_vm.ha_cp
terraform apply -target=proxmox_virtual_environment_vm.ha_cp

Two minutes and 45 seconds later, the VM was up and SSH was answering. The BeeLinks were already running Ubuntu 24.04 from a previous setup session, so all three nodes were ready.

Step 2: Install K8s Prerequisites (Ansible)

This is the boring but critical part. Every node needs the same stack: containerd, kubeadm, kubelet, kubectl, plus kernel modules and sysctl tweaks for container networking.

I wrote a single Ansible playbook (ansible/playbook-ha-k8s.yml) that handles the entire cluster lifecycle. The prerequisite tasks run on all three nodes in parallel.

The inventory

ha_cluster:
  vars:
    ansible_user: ubuntu
    ha_vip: 192.168.1.222
    ha_vip_port: 6443
  children:
    ha_cluster_init:
      hosts:
        192.168.1.42:
          ha_interface: eth0
    ha_cluster_join:
      hosts:
        n1.localdomain:
          ha_interface: enp1s0
        n2.localdomain:
          ha_interface: enp1s0

The ha_interface is important because it's different on each type of hardware. The Proxmox VM uses eth0, the BeeLinks use enp1s0 (Realtek NICs). kube-vip needs to know which interface to bind the VIP to.

What the playbook does

Here's the condensed version. The full playbook is in the repo at ansible/playbook-ha-k8s.yml.

Kernel modules and sysctl:

- name: Load required kernel modules
  modprobe:
    name: "{{ item }}"
  loop: [overlay, br_netfilter]

- name: Set required sysctl params
  sysctl:
    name: "{{ item.key }}"
    value: "{{ item.value }}"
    sysctl_file: /etc/sysctl.d/k8s.conf
  loop:
    - { key: net.bridge.bridge-nf-call-iptables, value: "1" }
    - { key: net.bridge.bridge-nf-call-ip6tables, value: "1" }
    - { key: net.ipv4.ip_forward, value: "1" }

Install containerd from Docker's apt repo:

- name: Install containerd
  apt:
    name: containerd.io
    state: present
    update_cache: true

Install kubeadm, kubelet, kubectl from the K8s repo:

- name: Add Kubernetes apt repository
  copy:
    dest: /etc/apt/sources.list.d/kubernetes.list
    content: "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.35/deb/ /\n"

- name: Install kubeadm, kubelet, kubectl
  apt:
    name: [kubeadm, kubelet, kubectl]
    state: present
    update_cache: true

The containerd v2 gotcha

This one got me. containerd v2.2.2 (which is what Docker's apt repo gives you on Ubuntu 24.04) ships with a default config that disables the CRI plugin:

disabled_plugins = ["cri"]

CRI is literally the interface Kubernetes uses to talk to the container runtime. With it disabled, kubeadm init fails with a cryptic error about unknown service runtime.v1.RuntimeService. If you see that, check your containerd config. The fix:

- name: Enable CRI plugin (disabled by default in containerd v2)
  replace:
    path: /etc/containerd/config.toml
    regexp: 'disabled_plugins = \["cri"\]'
    replace: 'disabled_plugins = []'

You also need SystemdCgroup = true in the containerd config so the cgroup driver matches kubelet. The playbook handles both.

Step 3: Initialize the First Control Plane with kube-vip

This is the interesting part. In a multi-master K8s cluster, you need a stable endpoint for the API server that doesn't depend on any single node. If the API server on node 1 goes down, kubectl (and every kubelet) needs to automatically reach node 2 or 3 instead.

Why kube-vip

I already have a keepalived setup for my HA load balancer, so I considered reusing it. But that would create a circular dependency: the K8s API would depend on the LB stack, and if I ever moved the LB into K8s (unlikely, but possible), I'd have a chicken and egg problem.

kube-vip is purpose-built for this. It runs as a static pod on each control plane node and uses leader election to assign a floating virtual IP (VIP). Only the leader node holds the VIP. If it dies, another node wins the election and takes over, usually within seconds. No external dependencies. The K8s cluster is entirely self-contained.

My VIP: 192.168.1.222 (my HA LB uses .221, so this sits right next to it)

The kube-vip manifest

I use an Ansible template (ansible/templates/kube-vip.yaml.j2) that gets deployed to /etc/kubernetes/manifests/ as a static pod:

apiVersion: v1
kind: Pod
metadata:
  name: kube-vip
  namespace: kube-system
spec:
  containers:
  - args: [manager]
    env:
    - name: vip_arp
      value: "true"
    - name: port
      value: "6443"
    - name: vip_interface
      value: "{{ ha_interface }}"
    - name: address
      value: "{{ ha_vip }}"
    - name: cp_enable
      value: "true"
    - name: vip_leaderelection
      value: "true"
    - name: vip_leaseduration
      value: "5"
    - name: vip_renewdeadline
      value: "3"
    - name: vip_retryperiod
      value: "1"
    image: ghcr.io/kube-vip/kube-vip:v1.1.2
    securityContext:
      capabilities:
        add: [NET_ADMIN, NET_RAW, SYS_TIME]
    volumeMounts:
    - mountPath: /etc/kubernetes/admin.conf
      name: kubeconfig
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/{{ kube_vip_kubeconfig }}
    name: kubeconfig

The K8s 1.29+ bootstrap problem

Notice that kube_vip_kubeconfig variable? That's there because of a breaking change in Kubernetes 1.29. Before 1.29, admin.conf had full cluster-admin privileges from the moment kubeadm init started. kube-vip could mount it and immediately acquire the leader election lease.

Starting with 1.29, admin.conf doesn't get its ClusterRoleBinding until later in the bootstrap process. Instead, kubeadm creates a super-admin.conf with the old behavior. So the trick is:

1. Before kubeadm init: Deploy kube-vip with super-admin.conf
2. Run kubeadm init: kube-vip can authenticate and grab the VIP
3. After init succeeds: Switch kube-vip to admin.conf for ongoing operation

The playbook handles this automatically with two template deployments using different variables.

Running kubeadm init

- name: Initialize Kubernetes control plane
  command: >
    kubeadm init
      --control-plane-endpoint "{{ ha_vip }}:{{ ha_vip_port }}"
      --upload-certs
      --pod-network-cidr 10.244.0.0/16

The --control-plane-endpoint flag is what makes this an HA cluster. It tells kubeadm that the API server should be accessed through the VIP, not the node's own IP. Every kubelet and every kubeconfig will point to 192.168.1.222:6443.

--upload-certs encrypts and uploads the control plane certificates to a cluster secret so the joining nodes can download them. The encryption key expires in 2 hours, which is plenty of time for automated joining.

--pod-network-cidr 10.244.0.0/16 is what Flannel expects. Speaking of which:

- name: Install Flannel CNI
  command: kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

Step 4: Join the BeeLinks

Each joining node needs two things before running kubeadm join: the kube-vip manifest and the join command from the init node.

The playbook fetches fresh join credentials from the init node (in case tokens have rotated), deploys the kube-vip template, and runs the join:

- name: Join as control plane node
  command: >
    {{ hostvars[groups['ha_cluster_init'][0]]['ha_join_command'] }}
      --control-plane
      --certificate-key {{ hostvars[groups['ha_cluster_init'][0]]['ha_cert_key'] }}

The --control-plane flag is what differentiates a control plane join from a worker join. It downloads the certificates and sets up etcd, the API server, and the other control plane components.

The nodes join one at a time (serial: 1 in Ansible) because etcd membership changes require consensus. Trying to add two members simultaneously can cause quorum issues.

Step 5: Untaint and Verify

By default, kubeadm taints control plane nodes with NoSchedule so workloads only run on workers. Since all three of our nodes are both control plane and worker, we remove the taint:

- name: Remove NoSchedule taint from control plane nodes
  command: kubectl taint nodes {{ item }} node-role.kubernetes.io/control-plane:NoSchedule-
  loop: "{{ node_names.stdout.split() }}"

This is easily reversible. When more hardware arrives, we can re-add the taint and let the control plane nodes focus exclusively on cluster management.

The Result

$ kubectl get nodes -o wide
NAME    STATUS   ROLES           AGE   VERSION   INTERNAL-IP     OS-IMAGE
ha-cp   Ready    control-plane   4m    v1.35.3   192.168.1.42    Ubuntu 24.04.4 LTS
n1      Ready    control-plane   2m    v1.35.3   192.168.1.40    Ubuntu 24.04.4 LTS
n2      Ready    control-plane   50s   v1.35.3   192.168.1.41    Ubuntu 24.04.4 LTS

Three nodes. Three etcd members. Three API servers. Three instances of kube-vip doing leader election. All pods healthy:

NAMESPACE      NAME                            READY   STATUS
kube-flannel   kube-flannel-ds-*               1/1     Running   (x3)
kube-system    coredns-*                       1/1     Running   (x2)
kube-system    etcd-*                          1/1     Running   (x3)
kube-system    kube-apiserver-*                1/1     Running   (x3)
kube-system    kube-controller-manager-*       1/1     Running   (x3)
kube-system    kube-scheduler-*                1/1     Running   (x3)
kube-system    kube-vip-*                      1/1     Running   (x3)
kube-system    kube-proxy-*                    1/1     Running   (x3)

The VIP responds:

$ curl -sk https://192.168.1.222:6443/healthz
ok

And the old cluster? Still running. I set up a separate kubeconfig (~/.kube/config-ha) so I can talk to either cluster explicitly:

# Default kubectl still points to the old cluster
kubectl get nodes
# NAME           STATUS   ROLES           VERSION
# k8s-master     Ready    control-plane   v1.31.14
# k8s-worker-1   Ready    <none>          v1.31.14
# k8s-worker-2   Ready    <none>          v1.31.14

# Explicit kubeconfig for new HA cluster
kubectl --kubeconfig ~/.kube/config-ha get nodes
# NAME    STATUS   ROLES           VERSION
# ha-cp   Ready    control-plane   v1.35.3
# n1      Ready    control-plane   v1.35.3
# n2      Ready    control-plane   v1.35.3

What I Learned

containerd v2 is sneaky. The default config disabling CRI is not documented in most K8s installation guides because they were written for containerd v1. If you're installing in 2025 or later, check your config.toml.

kube-vip's bootstrap dance is annoying but logical. The super-admin.conf workaround is the kind of thing that makes you question whether the juice is worth the squeeze, until you realize the alternative is an external load balancer just for the API server. One sed command during bootstrap is a fair trade.

Mixed hardware works fine. I expected weird edge cases from running VMs alongside bare metal. There weren't any. kubeadm doesn't care that one node is virtualized and two aren't. etcd doesn't care. The only difference was the network interface names (eth0 vs enp1s0), and that's why the Ansible inventory has per-host variables.

Serial joins matter. Joining two control plane nodes simultaneously can cause etcd to lose quorum during the membership change. The serial: 1 in the Ansible playbook makes the joins sequential. It's slower but safe.

What's Next (Part 2)

The cluster is running but empty. In Part 2, I'll:

• Install Traefik as the ingress controller
• Migrate workloads from the old single-master cluster (Vaultwarden, ArgoCD, MarketMind)
• Retarget the HA load balancer (Caddy) to route traffic to the new cluster
• Decommission the old cluster and reclaim the resources

The hard part is done. The easy part is next. Famous last words.

Featured Image Prompt

I used this prompt to generate the featured image. Three interconnected nodes forming a triangle, each node a different piece of hardware: one is a rack-mounted server (Proxmox), the other two are tiny BeeLink mini PCs. Glowing data streams flow between them in a circular pattern. A floating Kubernetes wheel logo hovers above the triangle. The center of the triangle shows "HA" in holographic text. Dark moody homelab aesthetic with deep blue and electric purple accent lighting, circuit board patterns visible in the background, photorealistic 3D render.