Menu
Article ansible

Entire Homelab Dies Because One Machine Reboots: Building an HA Load Balancer with Keepalived and Caddy

Emir Ibrahimbegovic

Emir Ibrahimbegovic

• 12 min read
Entire Homelab Dies Because One Machine Reboots: Building an HA Load Balancer with Keepalived and Caddy

Why I Did This

My Proxmox host rebooted, and everything went dark.

Vaultwarden, ArgoCD, my monitoring dashboard, the blog, Paperless. All of it. Every service in my homelab routed through a Kubernetes cluster running on Proxmox VMs, and the load balancer for that cluster was MetalLB, which also ran inside Kubernetes. So when Proxmox went down for a firmware update, the load balancer died with it, which meant the ingress died, which meant DNS entries pointing at the ingress went nowhere. A planned five-minute reboot turned into a "why is everything broken" moment for anyone in the house trying to use anything I host.

The irony was thick. I had a K8s cluster with multiple worker nodes, redundant pod replicas, health checks, the works. But none of that matters when your load balancer is a pod inside the same cluster it's supposed to be balancing. It's like putting the pilot's ejection seat inside the engine.

What MetalLB Actually Does (and Why It Wasn't Enough)

MetalLB is a bare-metal Kubernetes load balancer. It gives your services a real IP address on your LAN using ARP announcements, so you can hit 192.168.1.201 and reach your Traefik ingress controller. For a lot of homelabs, this is perfectly fine.

Here's the problem: MetalLB runs as pods inside K8s. If K8s goes down, MetalLB goes down, and that IP stops responding. There's no failover outside the cluster. Your DNS entries still point at 192.168.1.201, but nobody's home.

I also had services running outside K8s entirely. SiYuan (notes), Paperless (document management), and my Ghost blog all run as Docker containers on a separate Portainer host. They were routed through K8s using an external-services pattern (headless Service + Endpoints pointing at the Docker host IP). This worked, but it meant that even non-K8s services depended on K8s being alive. A notes app shouldn't go offline because Kubernetes is having a bad day.

And then there's Beszel (my monitoring dashboard) and Frigate (security cameras), both running natively on the Mac Mini M1 that has nothing to do with Proxmox or K8s. These could survive a Proxmox outage entirely, but I couldn't reach them because the routing went through the dead cluster.

The Plan

  1. Remove MetalLB from K8s (done)
  2. Expose Traefik via NodePort instead of LoadBalancer (done)
  3. Set up 3 LB nodes across different physical machines (done)
  4. Install keepalived for floating VIP (VRRP) (done)
  5. Install Caddy as the reverse proxy on each node (done)
  6. Ansible playbook for repeatable deployment (done)
  7. Update DNS entries to point at VIP (done)
  8. Update cloudflared to route through VIP (done)
  9. Auto-start Mac-hosted VMs on boot (done)

Choosing the Stack

Why Keepalived?

VRRP (Virtual Router Redundancy Protocol) is the simplest answer to "what if the machine holding the IP goes down?" Multiple nodes run keepalived, each with a priority. The highest priority healthy node becomes the master and owns the virtual IP (VIP). If the master dies, the next highest takes over. Usually within a second.

No coordination service. No consensus algorithm. No etcd. Just VRRP advertisements over multicast, every second. The nodes don't even need to know about each other's existence. They just broadcast "I'm alive and my priority is X" and the protocol handles the rest.

Why Caddy?

I needed a reverse proxy on each LB node that could:

  • Terminate TLS with my private CA's wildcard certificate
  • Route by hostname to different backends
  • Health-check K8s workers and round-robin across them
  • Run with almost zero configuration

Caddy does all of this out of the box. The Caddyfile is readable by humans, TLS just works if you hand it a cert and key, and it has built-in health checking for upstreams. I already use Traefik inside K8s, but Caddy felt like the right tool for a standalone reverse proxy that should be as boring and reliable as possible.

I looked at HAProxy and Nginx too. HAProxy would work but the config syntax is verbose for what I need. Nginx would also work but I'd have to manually configure health checks and TLS in a way that Caddy just handles automatically. For three nodes running identical configs, simplicity wins.

Why Three Nodes?

Two nodes give you failover. Three give you failover without anxiety. If one of the Mac Minis is off for maintenance and then Proxmox reboots, you still have a working load balancer. The cost is minimal: each LB node runs Ubuntu with 512MB RAM and 1 CPU core. Three lightweight VMs spread across three physical machines.

Node IP Physical Host Priority Role
lb-m4 192.168.1.22 Mac Mini M4 Pro (UTM VM) 100 Default master
lb-proxmox 192.168.1.20 Proxmox (QEMU VM) 90 Backup
lb-m1 192.168.1.23 Mac Mini M1 (UTM VM) 80 Backup

VIP: 192.168.1.221/24 (outside the DHCP range)

The Mac Mini M4 Pro gets priority 100 because it's the beefy daily driver that's always on. The Proxmox VM gets 90 because it's the most likely to go down during maintenance. The Mac Mini M1 gets 80 as the last resort.

The Architecture

Here's what the traffic flow looks like now:

                                    LAN devices
                                  (laptops, phones,
                                   other machines)
                                        |
Internet --> Cloudflare                 |
              --> cloudflared           |
                    |                   |
                    |    *.localdomain   |
                    |    DNS resolves    |
                    |    to VIP         |
                    v                   v
              HA Load Balancer (VIP 192.168.1.221)
              keepalived + Caddy, 3 nodes
              |
              +-- K8s services (vault, argocd, market-mind)
              |     Caddy --> Traefik NodePort 30443
              |     Round-robin across K8s workers
              |
              +-- Portainer services (siyuan, paperless, errbit)
              |     Caddy --> direct to 192.168.1.2
              |
              +-- Mac Mini M1 services (metrics, frigate)
              |     Caddy --> direct to 192.168.1.5
              |     Fully HA: survives Proxmox outage
              |
              +-- Ghost blog (:80)
                    cloudflared --> Caddy --> 192.168.1.2:2368

Most traffic isn't coming from the internet. It's coming from inside the house. Laptops opening Vaultwarden to grab a password. Phones checking Frigate camera feeds. My dev machine hitting ArgoCD or the monitoring dashboard. Other services calling each other over *.localdomain hostnames. The router resolves all *.localdomain DNS entries to the VIP (192.168.1.221), so every device on the network hits Caddy the same way an external request would, just without the Cloudflare hop. Public traffic (the blog, MarketMind) comes in through Cloudflare and cloudflared, but that's a small fraction of the total load.

The key insight: Caddy doesn't just proxy to K8s. It routes everything by hostname. K8s services go through Traefik's NodePort, Docker services go directly to the Portainer host, and Mac Mini M1 services go directly to that machine. Each class of service has a different failure domain. If Proxmox goes down, you lose K8s and Portainer services, but metrics and Frigate stay up because they live on the Mac Mini M1, a completely separate physical machine.

Setting Up the Nodes

Proxmox VM (the easy one)

This was the straightforward node. Clone an Ubuntu template, assign a static IP via cloud-init, SSH in, done. I use Terraform to provision VMs on Proxmox, so lb-proxmox is defined as infrastructure-as-code alongside the K8s nodes:

resource "proxmox_vm_qemu" "lb_node" {
  name        = "lb-node"
  target_node = "proxmox"
  vmid        = 105
  clone       = "ubuntu-template"
  full_clone  = true
  agent       = 1
  onboot      = true
  cores       = 1
  memory      = 512

  disk {
    type    = "disk"
    slot    = "scsi0"
    size    = "8G"
    storage = "local-lvm"
  }

  network {
    id     = 0
    model  = "virtio"
    bridge = "vmbr0"
  }

  ipconfig0  = "ip=192.168.1.20/24,gw=192.168.1.1"
  nameserver = "192.168.1.1"
  ciuser     = "ubuntu"
  sshkeys    = file("~/.ssh/my_key.pub")
}

Mac UTM VMs (the adventure)

UTM is a QEMU-based hypervisor for Apple Silicon. It runs ARM64 Linux VMs natively, which is exactly what I needed for the Mac Mini M4 Pro and Mac Mini M1 nodes.

The catch: UTM doesn't have cloud-init, Terraform, or any automation story. You install Ubuntu from an ISO, manually configure networking, and SSH in (scp your public key into authorized_keys). For the Mac Mini M1, which runs headless in a closet, I had to do the whole install over screen sharing, then set up a LaunchAgent to auto-start the VM on boot.

Bridged networking is critical. The VM needs to be on the same L2 network as the other LB nodes for VRRP to work. In UTM, this means selecting the wired Ethernet adapter (not WiFi) as the bridged interface. On Mac, en0 is wired and en1 is WiFi. Getting this backwards means your VM gets an IP but VRRP advertisements never reach the other nodes.

Auto-start on boot uses a LaunchAgent plist that waits 15 seconds (for UTM to finish launching) then runs utmctl start:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>com.utm.autostart-m4-lb</string>
  <key>ProgramArguments</key>
  <array>
    <string>/bin/bash</string>
    <string>-c</string>
    <string>sleep 15 &amp;&amp; /Applications/UTM.app/Contents/MacOS/utmctl start m4-lb</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
</dict>
</plist>

Drop this in ~/Library/LaunchAgents/, enable auto-login on the Mac, and disable FileVault. The VM comes up about 30 seconds after boot. Not instant, but keepalived handles the gap. The other nodes own the VIP until this one joins.

The Configs

Keepalived (VRRP)

Each node runs the same keepalived config, templated by Ansible with per-node priority and interface:

vrrp_instance VI_1 {
  state BACKUP
  interface {{ lb_interface }}
  virtual_router_id 51
  priority {{ lb_priority }}
  advert_int 1

  authentication {
    auth_type PASS
    auth_pass {{ lb_vrrp_pass }}
  }

  virtual_ipaddress {
    192.168.1.221/24 dev {{ lb_interface }}
  }
}

Every node starts as BACKUP. The one with the highest priority wins the election and becomes master. advert_int 1 means nodes advertise every second. If the master misses three advertisements (3 seconds), failover happens. In practice, I see sub-second failover in my testing.

All nodes use the same virtual_router_id (51) and authentication password so they recognize each other as part of the same VRRP group. The interface varies: eth0 on the Proxmox VM, enp0s1 on the UTM VMs.

Caddy (Reverse Proxy)

This is the fun part. The Caddyfile is identical on all three nodes, templated once and deployed everywhere:

# Reusable snippet for K8s services via Traefik NodePort
(k8s_backend) {
  reverse_proxy 192.168.1.11:30443 192.168.1.12:30443 {
    transport http {
      tls
      tls_insecure_skip_verify
    }
    lb_policy round_robin
    health_interval 10s
    health_timeout 5s
  }
}

# K8s services
vault.localdomain {
  tls /etc/caddy/certs/wildcard.crt /etc/caddy/certs/wildcard.key
  import k8s_backend
}

argocd.localdomain {
  tls /etc/caddy/certs/wildcard.crt /etc/caddy/certs/wildcard.key
  import k8s_backend
}

market-mind.localdomain {
  tls /etc/caddy/certs/wildcard.crt /etc/caddy/certs/wildcard.key
  import k8s_backend
}

# Docker/Portainer services (direct, no K8s involved)
siyuan.localdomain {
  tls /etc/caddy/certs/wildcard.crt /etc/caddy/certs/wildcard.key
  reverse_proxy 192.168.1.2:6806
}

paperless.localdomain {
  tls /etc/caddy/certs/wildcard.crt /etc/caddy/certs/wildcard.key
  reverse_proxy 192.168.1.2:28981
}

# Fully HA services (Mac Mini M1, survives Proxmox outage)
metrics.localdomain {
  tls /etc/caddy/certs/wildcard.crt /etc/caddy/certs/wildcard.key
  reverse_proxy 192.168.1.5:8090
}

# Public HTTP (Ghost blog via cloudflared)
:80 {
  reverse_proxy 192.168.1.2:2368 {
    header_up X-Forwarded-Proto https
  }
}

A few things worth noting:

The k8s_backend snippet is a Caddy named snippet. Every K8s service imports it, so adding a new one is a single block with import k8s_backend. The snippet round-robins across both K8s worker nodes on port 30443 (Traefik's NodePort) with health checks every 10 seconds. If a worker goes down, Caddy stops routing to it automatically.

tls_insecure_skip_verify is needed because Traefik presents a self-signed certificate on the NodePort. Caddy is talking to Traefik over the LAN, so verifying Traefik's cert isn't necessary. The wildcard cert from step-ca is what clients (browsers) see.

The X-Forwarded-Proto: https header on port 80 prevents Ghost from going into an infinite redirect loop. Ghost sees HTTP from Caddy, tries to redirect to HTTPS, Cloudflare sends it back as HTTP, Ghost redirects again. Setting the header tells Ghost "the client is already on HTTPS, stop redirecting."

metrics.localdomain points directly at the Mac Mini M1. This service has zero dependency on Proxmox, K8s, or Docker. If Proxmox burns to the ground, my monitoring dashboard stays up. That felt important.

Traefik: LoadBalancer to NodePort

With Caddy handling ingress, Traefik no longer needs a MetalLB LoadBalancer IP. Instead, it exposes a NodePort service that Caddy routes to:

apiVersion: v1
kind: Service
metadata:
  name: traefik-nodeport
  namespace: traefik
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: traefik
  ports:
    - name: web
      port: 80
      targetPort: web
      nodePort: 30080
    - name: websecure
      port: 443
      targetPort: websecure
      nodePort: 30443

Traefik still does all the K8s-internal routing (IngressRoutes, middleware, TLS termination for K8s services). It just doesn't own a LAN IP anymore. Caddy owns the LAN IP (via the VIP) and forwards to Traefik on the NodePort.

I left the old LoadBalancer service in place. It shows <pending> now that MetalLB is gone, but it's harmless and removing it would require updating the Helm values.

Ansible: Making It Repeatable

Three nodes running identical configs is the perfect Ansible use case. The entire deployment is a single playbook:

- name: Deploy HA load balancer (keepalived + Caddy)
  hosts: lb_nodes
  become: true
  tasks:
    - name: Install keepalived and caddy
      ansible.builtin.apt:
        name: [keepalived, caddy]
        state: present
        update_cache: true

    - name: Create cert directory
      ansible.builtin.file:
        path: /etc/caddy/certs
        state: directory
        owner: caddy
        group: caddy
        mode: "0750"

    - name: Copy wildcard TLS certificate
      ansible.builtin.copy:
        src: "{{ lb_cert_src }}"
        dest: /etc/caddy/certs/wildcard.crt
        owner: caddy
        group: caddy
        mode: "0640"
      notify: Reload caddy

    - name: Copy wildcard TLS key
      ansible.builtin.copy:
        src: "{{ lb_key_src }}"
        dest: /etc/caddy/certs/wildcard.key
        owner: caddy
        group: caddy
        mode: "0640"
      notify: Reload caddy

    - name: Template Caddyfile
      ansible.builtin.template:
        src: templates/Caddyfile.j2
        dest: /etc/caddy/Caddyfile
      notify: Reload caddy

    - name: Template keepalived config
      ansible.builtin.template:
        src: templates/keepalived.conf.j2
        dest: /etc/keepalived/keepalived.conf
      notify: Restart keepalived

    - name: Enable and start services
      ansible.builtin.systemd:
        name: "{{ item }}"
        enabled: true
        state: started
      loop: [caddy, keepalived]

  handlers:
    - name: Reload caddy
      ansible.builtin.systemd:
        name: caddy
        state: reloaded

    - name: Restart keepalived
      ansible.builtin.systemd:
        name: keepalived
        state: restarted

The inventory defines per-node variables:

lb_nodes:
  hosts:
    192.168.1.20:
      ansible_user: ubuntu
      lb_priority: 90     # Proxmox VM
      lb_interface: eth0
    192.168.1.23:
      ansible_user: ubuntu
      lb_priority: 80     # Mac Mini M1 UTM VM
      lb_interface: enp0s1
    192.168.1.22:
      ansible_user: ubuntu
      lb_priority: 100    # Mac Mini M4 Pro UTM VM
      lb_interface: enp0s1

One command deploys or updates all three nodes:

ansible-playbook -i inventory.yml playbook-ha-lb.yml --extra-vars '@lb-keys.yml'

The lb-keys.yml file (gitignored) contains the VRRP shared secret and paths to the TLS cert/key from step-ca. The cert gets copied from my Mac (which acts as an SCP relay between step-ca and the LB nodes).

What Changed (The Migration)

The actual cutover from MetalLB to the HA LB was surprisingly smooth. Here's what happened:

  1. Applied the NodePort service to K8s so Traefik is reachable on port 30443 on every worker
  2. Ran the Ansible playbook to deploy keepalived + Caddy on all three nodes
  3. Updated router DNS to point all *.localdomain hostnames at 192.168.1.221 (the VIP) instead of 192.168.1.201 (old MetalLB IP)
  4. Updated cloudflared config to route through the VIP instead of directly to Traefik
  5. Removed MetalLB from the K8s cluster (optional, but why keep it around)
  6. Removed external-services K8s manifests for Portainer services (Caddy routes to them directly now, no need for the headless Service + Endpoints hack)

The only downtime was the DNS change, which propagates instantly on my router since it's the authoritative DNS for .localdomain. Total switchover: about 30 seconds.

Failure Scenarios

This is the part I actually tested. Repeatedly. By pulling network cables and watching what happened.

Proxmox reboots (the original problem):
K8s services go down (expected, VMs are on Proxmox), lb-proxmox goes down. But lb-m4 or lb-m1 keeps the VIP alive. When Proxmox comes back, lb-proxmox rejoins automatically and starts receiving traffic again. Metrics and Frigate never went down because they live on the Mac Mini M1.

Mac Mini M4 Pro goes offline (highest priority node):
lb-proxmox (priority 90) takes over the VIP within ~1 second. All services continue working. When lb-m4 comes back, it reclaims the VIP (preemption is enabled by default in keepalived).

Two nodes down simultaneously:
The remaining node owns the VIP alone. Services routed to the surviving backends still work. This is why three nodes across three physical machines matters. The odds of all three dying at once are vanishingly small.

Complete power outage:
Everything dies. But when power returns, the Mac Minis auto-login, UTM launches, the LaunchAgents start the VMs, keepalived elects a master, and the VIP comes up. No manual intervention. I tested this by flipping the breaker. It took about 90 seconds from power-on to VIP-active.

What I Learned

The biggest lesson (That I already knew and applied only at the job): your load balancer can't live inside the thing it's balancing. This seems obvious in retrospect, but MetalLB makes it so easy to set up that you don't think about it until it bites you. For production Kubernetes clusters backed by cloud providers, this isn't an issue because the cloud LB is external. For homelabs on bare metal, you have to build that external layer yourself.

Keepalived is ridiculously simple for what it does. The entire config is 16 lines. VRRP is a solved protocol from the 1990s and it just works. No service discovery, no leader election complexity, no distributed state. Three daemons yelling their priority over the network every second. The loudest healthy one wins. Beautiful.

Caddy replaced an entire Kubernetes routing layer (MetalLB + external-services pattern) with a single config file. I went from "headless Service with manually maintained Endpoints pointing at a LAN IP, routed through an IngressRoute on Traefik that I need MetalLB to expose" to "reverse_proxy 192.168.1.2:6806". Seven lines of YAML became one line of Caddyfile.

Spreading nodes across physically separate machines is the real HA move. If all three VMs were on Proxmox, this entire exercise would have been pointless. The Mac Minis are the insurance policy. They're low-power, always-on machines that don't share a failure domain with the hypervisor. One of them even hosts services that are completely Proxmox-independent. That's the setup I should have had from day one.

And UTM on Apple Silicon is surprisingly capable for running lightweight Linux VMs. It's not Proxmox, it's not VMware, and it has zero automation story. But for a 512MB Ubuntu VM running keepalived and Caddy, it's perfectly adequate. Just budget extra time for the initial setup if the Mac is headless.

Share