Making Cloudflare Tunnel Actually Highly Available

I've been running emir.fyi and a handful of other services through a single Cloudflare Tunnel for months. Worked fine. Never thought about it.

Then I started setting up a new domain example.com as a dry-run for another domain that's about to carry lead-gen traffic for my wife's property management business. Halfway through, I clocked the obvious thing: everything public on this homelab rides through a single cloudflared VM at 192.168.1.15. One VM. One kernel. One shot.

The HA load balancer is HA. The K8s control plane is HA. The thing routing external traffic to both of them is not. Months of building redundancy downstream, and the funnel into the whole homelab is a single box I hadn't thought about since I spun it up.

Time to fix that before a real domain depends on it.

How Cloudflare Tunnel HA actually works

You don't need a load balancer in front of cloudflared. Cloudflare already has one, its entire edge network. The trick is that multiple cloudflared processes can run simultaneously, all authenticated against the same tunnel ID. Each one opens 4 outbound QUIC connections to Cloudflare's edge. Cloudflare sees N connectors on one tunnel, load-balances traffic across them, and drops any connector that disconnects. Sub-second failover. Zero coordination between the replicas, they don't know about each other.

The only invariants are: same tunnel ID, identical credentials, identical ingress rules on every replica. Drift any of those three and failover can route a request to a connector that can't handle it.

The design

Three replicas, one tunnel:

The BeeLinks are already pulling dual duty, they run K8s control-plane workloads alongside everything else. Adding cloudflared as another systemd service on each is basically free (the whole binary is ~30MB resident). The trade off: if a BeeLink dies, I lose both a K8s CP member and a cloudflared replica at once. Still within both quorums, but the blast radius per node failure gets wider.

I considered running cloudflared as a Kubernetes Deployment instead. Clean on paper, replicas, rolling updates, anti affinity. But putting the thing that fronts public traffic inside the cluster it's supposed to be fronting felt incestuous for a first pass, and the failure mode I most wanted to validate is "a single node dies," which is easier to reason about with dedicated processes on dedicated hosts. Saving that migration for later.

Capturing the existing config

Before writing any playbook, I pulled the current state off the primary VM. If my rendered config didn't match what was already running on 192.168.1.15 byte-for-byte, a future Ansible run could silently rewrite it and break emir.fyi.

ansible -i inventory.yml cloudflared.localdomain -m command -a "cat /etc/cloudflared/config.yml" -b

tunnel: homelab
credentials-file: /etc/cloudflared/<tunnel-uuid>.json

ingress:
  - hostname: emir.fyi
    service: http://192.168.1.221:80

  - hostname: www.emir.fyi
    service: http://192.168.1.221:80

  - service: http_status:404

The credentials JSON I fetched with Ansible's fetch module straight to disk, silent copy, never touched stdout, lands in ansible/files/cloudflared-credentials.json (gitignored). This is the file every replica needs to be an identity-equivalent member of the tunnel.

The playbook

Stock Ansible modules only, no extra collections. Installs the Cloudflare apt repo, installs cloudflared, drops credentials + config + systemd unit, enables and starts the service. Every replica ends up with an identical /etc/cloudflared/ layout.

Key snippets from playbook-cloudflared.yml:

- name: Add Cloudflare apt repository
  apt_repository:
    # Cloudflare's repo uses "any" serves packages for all Debian/Ubuntu releases
    repo: "deb [signed-by=/usr/share/keyrings/cloudflare-main.gpg] https://pkg.cloudflare.com/cloudflared any main"
    state: present
    filename: cloudflared

- name: Deploy tunnel credentials
  copy:
    src: files/cloudflared-credentials.json
    dest: "/etc/cloudflared/{{ cloudflared_tunnel_id }}.json"
    owner: root
    group: root
    mode: "0400"
  notify: Restart cloudflared

One small gotcha caught on the first dry run: I had initialized the apt-repo line with {{ ansible_distribution_release }} (which resolves to noble on 24.04). But Cloudflare's repo uses a single generic pool called any. Check-mode showed one line of drift on the existing VM, which would have silently rewritten its sources.list.d entry on apply. Five-minute fix; a useful reminder that "canonical" Debian conventions aren't universal.

Rollout

Four phases, each verified before the next:

1. Dry-run against 192.168.1.15: --check --diff --limit cloudflared.localdomain had to be byte-identical no op. ok=10 changed=0.
2. Apply to n1 + n2 only (--limit n1.localdomain,n2.localdomain): fresh install, cert drop, service start. Tunnel went from 1 connector to 3 in the Cloudflare dashboard.
3. Apply to 192.168.1.15: real apply of the no-op diff to prove uniform GitOps management.
4. Rolling upgrade to latest: ansible cloudflared_hosts -m apt -a "name=cloudflared state=latest update_cache=yes" --forks 1. The critical flag is --forks 1 forces serial execution so the tunnel never drops below 2 healthy connectors. Upgraded 192.168.1.15 from 2026.2.0 to 2026.3.0 with zero downtime.

Throughout all four phases, I kept a curl https://emir.fyi loop running on my laptop. Every request came back HTTP 200, sub-200ms. No dropped requests.

The DNS plot twist

The plan for example.com was to run cloudflared tunnel route dns homelab example.com and have the CNAME materialize on the example.com zone. That's not what happened:

INF Added CNAME example.com.emir.fyi ...

It created example.com as a subdomain of emir.fyi. Wrong zone entirely.

Root cause: cloudflared's CLI authenticates against Cloudflare using ~/.cloudflared/cert.pem the login cert. I had assumed, and Cloudflare's docs kind of implied, that one cert.pem covers your whole account. The reality turns out to be different. I decoded the API token embedded in the cert and hit Cloudflare's /zones endpoint with it:

{
  "result": [
    { "id": "3cbc...", "name": "emir.fyi", ... }
  ],
  "result_info": { "count": 1, "total_count": 1 }
}

One zone. The embedded token has DNS:Edit on exactly one zone, whichever one you picked during cloudflared tunnel login. When you then ask the CLI to manage DNS for a different domain, it silently treats that domain as a subdomain of the one authorized zone. You don't get an error, you get a wrong record.

I re-ran cloudflared tunnel login and carefully clicked the example.com zone in the browser consent flow. Same result, the cert still had zoneID: 3cbc... (emir.fyi). The consent UI might have single-selected, or defaulted to the first zone, or there's a bug. Either way, cert.pem is structurally single-zone. You can juggle multiple cert files and switch via TUNNEL_ORIGIN_CERT=, but that's fragile.

The correct fix is to stop using cert.pem for DNS at all and use a scoped API token instead. Cloudflare's dashboard lets you mint a token with specific zone-level permissions across multiple zones at once. I created one covering emir.fyi, example.com, and eviasa.com (the next one up), with Zone:Read + DNS:Edit on each. Token stored in ansible/cloudflare-keys.yml (gitignored), following the same pattern as the existing lb-keys.yml.

Then I wrote playbook-cloudflare-dns.yml, runs on localhost, hits Cloudflare's API with the uri module (no extra collections required), and idempotently ensures a given list of hostnames has a proxied CNAME pointing to the tunnel. Key quirk: if the domain was previously on another host, you'll have leftover A records at the apex that conflict with a CNAME. The playbook detects and deletes them automatically, preserving MX/TXT/NS/etc. at the same name.

The extra work to build the token-based playbook paid off within an hour. When eviasa.com migrates to this tunnel, it'll be two new lines in cloudflared_cnames and a playbook run. No new auth, no consent flow, no accidents.

Verifying HA for real

The whole point of this exercise. I put the example.com apex behind a traefik/whoami container on Portainer, a tiny service that dumps request info including X-Forwarded-For, which tells me which cloudflared replica handled the request. Then I stopped each replica in turn and hammered the domain.

24 requests during simulated outages. Zero failures. Failover happened the moment each systemctl stop completed no sleep, no waiting. Cloudflare's edge notices a disconnected connector immediately and stops sending it traffic.

Interesting side note: Cloudflare's LB is not round robin. It clearly has preferences (n2 served the plurality of requests when all three were up), probably driven by connection latency per-PoP. Worth knowing if you ever need to debug uneven distribution, it's a feature, not a bug.

What I'd do differently

Two things.

First, I should have inspected cert.pem before I trusted what the docs said about it. One curl against /user/tokens/verify with the embedded token would have told me "this is scoped to one zone" immediately, not after two failed tunnel route dns commands and a cleanup on the emir.fyi zone. Empirical verification beats documentation every time, especially with CLIs that silently degrade instead of erroring.

Second, I went in thinking the hard part would be the rollout and installing cloudflared correctly on two new hosts, syncing credentials, making sure the service came up. That was thirty minutes. The hard part turned out to be the DNS glue, which I hadn't planned for at all. Worth remembering that the interesting failures in infra work almost never live where you expect them to.

The actual payoff

example.com is the dry-run. The real domain goes up next, a lead-gen property-management site where every hour of downtime is lost revenue. This setup means when it comes time to wire that one through, the entire infra path is already proven. Three cloudflared replicas survive any single failure. The DNS automation works across multiple zones. The Caddy routing scales to any new hostname via one matcher line. The whoami container gives me a sanity backend for smoke-testing any new domain before its real service exists.

All I'll need to do is add a couple of entries to two playbook files and re-run. Which is exactly what this kind of infrastructure work should feel like.

Featured Image Prompt

I used this prompt to generate the featured image. Three parallel glowing tunnels arcing between a homelab server rack on one side and a stylized Cloudflare cloud on the other. Each tunnel originates from a different piece of hardware: a rack-mounted VM and two tiny BeeLink mini PCs sitting on a shelf. Data packets flow bidirectionally through each tunnel as streams of light. One tunnel is momentarily dimmed mid-stream to show a replica failure, while the other two brighten and absorb its traffic in real time. Dark moody homelab aesthetic with Cloudflare orange and deep blue accent lighting, circuit board patterns subtly visible in the background, photorealistic 3D render.