![\"\"](\"https://emir.fyi/content/images/2026/04/tree-camera-1.webp\")
I wanted an offsite backup machine. That was the whole idea. Put a small box at a family member's house, hang a 12TB external drive off it, push backups to it over WireGuard, and finally stop pretending that \"backed up to the NAS in the same room\" counts as a backup strategy. Fire, theft, or a bad afternoon with a ransomware payload and the entire thing evaporates at once.
\nI did not want to buy hardware for it. I had two BeeLink mini PCs sitting in the rack, both of them part of the high availability Kubernetes cluster I built across mixed hardware, and one of them, I was fairly sure, was mostly idle. I remembered giving it a couple of jobs. Kubernetes something. Maybe a load balancer? I would go look, move whatever it was doing somewhere else, and reclaim the box.
\nReader, it was not mostly idle.
\nMy mental model of that node was \"Kubernetes control plane, a Postgres replica, maybe a load balancer node.\" Three jobs, two of which I could probably shuffle elsewhere in an afternoon. I checked it properly instead of trusting my memory, which is the first thing that went right.
\nkubectl get pods -A -o wide --field-selector spec.nodeName=n1\nThe real list:
\n| What | \nDetail | \n
|---|---|
| Control plane | \netcd, apiserver, controller manager, scheduler | \n
| kube-vip | \nserves the API VIP | \n
| Postgres | \none CloudNativePG instance | \n
| CoreDNS | \none of exactly two replicas | \n
| sealed-secrets | \nthe only replica | \n
| ArgoCD | \nserver and notifications controller | \n
| Apps | \ntwo workloads plus a web frontend | \n
| Host services | \none of my cloudflared tunnel connectors, monitoring agent | \n
The load balancer guess was wrong. My HA load balancer runs on three completely different machines. What I was thinking of was kube-vip, which serves the Kubernetes API VIP, a different VIP entirely from the one Caddy floats for services. Two VIPs, two purposes, one confused homelabber.
Nine jobs, not three. And note what is in that list: the only replica of sealed-secrets, one of exactly two CoreDNS replicas, a third of my tunnel connectors. This was not a spare box with a cluster membership card. It was a working node carrying a slice of nearly every service I run, and I had it filed in my head as \"mostly idle\" because the last time I thought about it, in April, it was.
\nMost of that list reschedules by itself, thankfully. Pods are cattle. The two that do not move on their own are the control plane role and the Postgres instance, and the control plane role is the one that turns this from \"drain a node\" into \"plan a migration.\"
\nHere is the thing I nearly got wrong, and it is the whole reason this post exists.
\netcd needs a majority to accept writes. Three members means a quorum of two, so you can lose one member and keep running. That is the entire point of running three.
\nTwo members means a quorum of two. You can lose zero.
\n| Members | \nQuorum | \nFailures tolerated | \n
|---|---|---|
| 3 | \n2 | \n1 | \n
| 2 | \n2 | \n0 | \n
| 1 | \n1 | \n0 | \n
A two member etcd cluster is not \"slightly less redundant than three.\" It is worse than a single node in a specific and nasty way: you now have two machines that can each independently take the whole cluster down, instead of one. You have doubled your failure surface and kept zero tolerance. It is the worst square on the board.
\nSo \"just remove the node and run on two until I get around to it\" was never an option. The replacement had to exist first.
\nThe sequence that works is three, then four, then three:
\n| Stage | \nMembers | \nQuorum | \nTolerates | \n
|---|---|---|---|
| Today | \n3 | \n2 | \n1 | \n
| New node joined | \n4 | \n3 | \n1 | \n
| Old node removed | \n3 | \n2 | \n1 | \n
Four members feels wasteful and is a bit odd (an even number buys you nothing over three), but it is a waypoint, not a destination. Crucially, tolerance never drops below where it started. At no point during the migration was the cluster less resilient than the day before.
\nI had two BeeLinks and assumed they were interchangeable. They were not, and the thing that decided it was storage.
\nBoth nodes use local-path provisioner volumes, which are node local disk with a hard node affinity baked into the PersistentVolume. A pod can reschedule anywhere. Its data cannot follow.
kubectl get pv -o json | jq -r '...'\n\npg-ha-1 -> node=n2\npg-ha-2 -> node=n1\npg-ha-3 -> node=ha-cp\nvaultwarden-data -> node=n2\nThere it is. My password vault lives on a 1GiB local-path volume pinned to n2. Retiring n2 would mean hand migrating that data or losing it. Retiring n1 means losing a Postgres replica, which the operator rebuilds from the primary automatically and without me being clever.
\nSo n1 it was, decided by a single line of PersistentVolume node affinity rather than by my vague sense of which box was busier.
\nThat discovery had a bonus prize attached, which is that I now know my password vault is sitting on one node's local disk with no backup anywhere. Small, irreplaceable, and the least protected thing I own. It went straight onto the TODO list, ahead of the photos.
\nThe replacement is a VM on my second hypervisor, which was running one Home Assistant VM and 6GB of its 62GB of RAM. Embarrassing amounts of headroom.
\nThe VM itself is Terraform, cloned from the Ubuntu template:
\nresource \"proxmox_virtual_environment_vm\" \"n3\" {\n name = \"n3\"\n node_name = \"hypervisor2\"\n vm_id = 141\n\n clone { vm_id = 9000 }\n cpu { cores = 4 }\n memory { dedicated = 16384 }\n disk { interface = \"scsi0\"; size = 150; datastore_id = \"local-lvm\" }\n\n initialization {\n ip_config { ipv4 { address = \"192.168.1.43/24\"; gateway = \"192.168.1.1\" } }\n }\n}\nThat disk is 150GB for a reason. A Postgres volume is going to land on this node, and I would rather size it once than resize it under pressure later.
\nMy cluster runs Kubernetes 1.35.3. My Ansible playbook installed the packages like this:
\n- name: Install kubeadm, kubelet, kubectl\n apt:\n name: [kubeadm, kubelet, kubectl]\n state: present\nNo version. So it installs whatever the repository is offering today. I checked before running it:
\nInstalled: 1.35.3-1.1\nCandidate: 1.35.8-1.1\nThe repo had drifted five patch releases ahead of my cluster while I was not looking. Unpinned, my brand new control plane node would have joined at 1.35.8 alongside three peers on 1.35.3. It probably would have worked. Patch skew inside a minor version usually does. But \"probably fine\" is not a thing I want to discover about my control plane at 11pm, and more to the point it would have been an accidental upgrade of one node, which is the kind of drift that is invisible until it is not.
\nThe fix is boring and permanent:
\n- name: Install kubeadm, kubelet, kubectl (pinned)\n apt:\n name:\n - \"kubeadm={{ k8s_package_version }}\"\n - \"kubelet={{ k8s_package_version }}\"\n - \"kubectl={{ k8s_package_version }}\"\n state: present\n allow_downgrade: true\nThe version lives in the inventory next to the cluster definition, where you bump it deliberately as an upgrade rather than absorbing it by accident.
\nNone of these were in my plan. All three are now written into the runbook, which is the actual product of an exercise like this.
\nMy playbook has a final play that removes the NoSchedule taint from control plane nodes, because in a three node homelab cluster every node has to run workloads. That play carries its own tag. I ran the join with --tags join. You can see where this is going.
The node joined perfectly. Then I moved the Postgres replica, and it sat in Pending for five minutes:
0/4 nodes are available: 1 node(s) had untolerated taint(s),\n1 node(s) were unschedulable, 2 node(s) didn't match pod anti-affinity rules\nThat message is a complete story if you read it. One node cordoned (the one I was draining), two nodes already holding a Postgres instance so anti-affinity rules them out, and one node tainted. Nowhere to go. The new node was tainted because the untaint play never ran.
\nRunning the tags separately felt tidy and surgical. It was actually just skipping steps.
\nDraining n1 moved its pods onto the other nodes, and two of them landed on n3. Those two are an app I build myself, so its image comes from a registry running in my own house rather than from Docker Hub. They would not start:
\nFailed to pull image \"registry.lan/myapp:latest\":\n x509: certificate signed by unknown authority\nEverything else in the cluster pulls from public registries that Ubuntu already trusts, so those pods came up without complaint. Only my own images come from my own registry, and that registry serves HTTPS with a certificate from my own internal certificate authority. Every other machine here has trusted that CA for as long as I can remember. The node I had built an hour earlier had never heard of it.
\nOne playbook fixed that, and it fixes it for every internal service at once rather than just the registry:
\nansible-playbook playbook-step-ca-trust.yml --limit n3\nThen it kept failing. Same error, character for character. containerd reads the system trust store when it starts, and it had been running for an hour, so the certificate was sitting on disk where it could not see it. A restart and the pull went straight through.
\nAn error message that does not change after you have fixed its cause is a genuinely nasty way to lose ten minutes.
\nThis is the one with real blast radius, and the one I nearly did not find.
\nMy Caddy load balancer round robins Kubernetes traffic across the control plane nodes:
\nreverse_proxy 192.168.1.40:30443 192.168.1.41:30443 192.168.1.42:30443 {\n lb_policy round_robin\n health_interval 10s\n health_timeout 5s\n }\nThat first address is the node I had just removed from the cluster. A third of my upstream slots pointed at a machine that no longer served the port.
\nEvery service still returned 200. Every single one. Because Caddy health checks the backends every ten seconds, noticed the dead one, and quietly routed around it. The system healed over the mistake so smoothly that nothing in my monitoring, and nothing in my browser, gave me any indication that my three way load balancing had silently become two way.
\nThat is the failure mode I find genuinely unsettling. Not the outage that pages you. The degradation that works.
\nI only caught it because I went looking through the docs for stale references to the old node's address and found it sitting in an Ansible template. If I had trusted \"everything returns 200,\" it would still be there, and I would have found out the next time one of the two surviving backends went down.
\nConfig that references cluster membership is state, and it drifts. The health check is a safety net, not a substitute for the config being right.
\n| \n | Before | \nAfter | \n
|---|---|---|
| Control plane | \nVM, BeeLink, BeeLink | \nVM, BeeLink, VM on the second hypervisor | \n
| Physical hosts | \n2 | \n3 | \n
| etcd members | \n3 | \n3, and never fewer during the swap | \n
| Postgres | \n3/3 | \n3/3, replica rebuilt on the new node | \n
| Tunnel connectors | \n3 | \n3, and never fewer than 2 during the swap | \n
The failure domain spread actually improved. Before, two of my three control plane nodes were BeeLinks sitting on the same shelf on the same power strip. Now the three members live on three separate physical machines. I did not set out to improve resilience, I set out to free up a box, but moving a node from \"one of two identical mini PCs\" to \"a completely different hypervisor\" is a genuine upgrade that fell out of the migration for free.
\nAnd the Postgres replica rebuild was the least dramatic part of the whole day. Delete the volume and the pod, and the operator notices an instance is missing, provisions a new volume on an available node, clones from the primary over streaming replication, and rejoins the quorum. My actual database is 922MB, so this took seconds. I watched a pod called pg-ha-4 appear on a machine that had existed for about an hour and start serving as a synchronous replica, and felt slightly redundant myself.
Nothing in a three node cluster is spare. Three is the minimum for a quorum. Every node is load bearing by definition, and \"I'll just take one back\" is not a thing you get to do without building its replacement first. If you want a spare machine, the number you need to start from is four.
\nAudit, do not remember. My recollection of that node's job was wrong in one place and incomplete in about six others. Thirty seconds of kubectl get pods --field-selector was worth more than everything I thought I knew about my own cluster.
The storage decides which node you can retire, not the CPU. I was thinking about which box was busier. The actual constraint was one line of node affinity on a PersistentVolume, pinning my password vault to a specific machine. Node local storage turns pods back into pets.
\nA fresh node is not a peer. It is missing every piece of state that accreted onto the others one incident at a time. Certificate trust, package pins, taints. Those are not documented anywhere because they were never a decision, they were just Tuesday. Building a new node is the only reliable way to find them, and writing them down afterwards is the only way to not rediscover them next time.
\nWatch out for the systems that heal over your mistakes. Health checks, retries, and automatic failover are exactly what you want, right up until they hide a configuration error so completely that you would never know except by reading the config. Redundancy that quietly becomes less redundant is the thing that gets you, because it feels identical to redundancy that works.
\nThe box is free now. The BIOS setting that makes it power itself back on after a blackout has to be sorted before it leaves the house, because at a family member's home there is nobody to press the button and a backup machine that silently stays off after a power blip is worse than no backup machine at all. That, and the offsite build itself, is the next post.
\nI used this prompt to generate the featured image.
\nA dimly lit home server rack photographed at a slight angle, three small mini PCs stacked on a shelf with soft blue and amber status LEDs glowing. One of the three machines is being lifted out by a pair of hands, and where it was there is a faint translucent blue wireframe outline of a replacement machine already materializing in the empty slot. Thin glowing cables connect all the machines, and the cables to the departing box are fading while the cables to the wireframe brighten. Shallow depth of field, moody dark background, cinematic teal and orange color grading, high detail, photorealistic, homelab aesthetic.
\n","comment_id":"6aa3620ab0ccab00016fcc52","feature_image":"https://emir.fyi/content/images/2026/09/fd499dc4-74aa-4d97-bb7d-92c839790b35.png","featured":false,"visibility":"public","created_at":"2026-09-10T22:06:02.000-04:00","updated_at":"2026-09-11T11:24:19.000-04:00","published_at":"2026-09-11T11:24:19.000-04:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/the-node-i-thought-was-spare/","excerpt":"What Started This\n\n\nI wanted an offsite backup machine. That was the whole idea. Put a small box at a family member's house, hang a 12TB external drive off it, push backups to it over WireGuard, and finally stop pretending that \"backed up to the NAS in the same room\" counts as a backup strategy. Fire, theft, or a bad afternoon with a ransomware payload and the entire thing evaporates at once.\n\n\nI did not want to buy hardware for it. I had two BeeLink mini PCs sitting in the rack, both of them pa","reading_time":10,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"6a9d7046ccab54000125377a","uuid":"5c79f38d-3746-4076-9eec-9a0620bfbdc8","title":"My Monitoring Hub Lived Inside the Thing It Was Watching","slug":"my-monitoring-hub-lived-inside-the-thing-it-was-watching","html":"Here is a fun little paradox I let sit in my homelab for way too long. My monitoring hub, the piece of software whose entire job is to tell me when something in the house has fallen over, was running on a Mac Mini in the house. Same power. Same internet. Same failure domain as everything it was supposed to be watching.
\nSo the day the power flickers or the ISP has a bad afternoon, here is what happens: the homelab goes dark, and the thing that is supposed to text me \"hey, the homelab went dark\" goes dark with it. The watchtower was inside the castle. If the castle burns, the tower does not get to send up a flare, because the tower is also on fire.
\nAnd sure enough this actually happened, while I was away for the world cup and I had no idea what's going on. Luckily I have great neighbors who came in and inspected the house. The issue was that my entire electrical circuit shut down, meaning my fancy ups power backup worked until it didn't. This isn't solution to that problem (which I've resolved in the meantime), but rather an additional enhancement towards remidiating a broken system.
\nI run Beszel for this, and I love it. It is tiny, it is fast, it does one job. But it was in the wrong place. What I wanted was a hub somewhere else entirely, on a box that stays up when my house does not, so it can be the one thing that survives to tell me the rest did not.
\nI already had a candidate. A little cloud VPS I stood up a while back to be a WireGuard exit node, sitting in a datacenter on someone else's power and someone else's uplink. It was doing one small job and had plenty of room for a second. Outside my blast radius, on infrastructure I trust more than my own breaker panel. Perfect.
\nThis is the story of moving the hub there. It went well, right up until the moment I texted myself that fourteen services were down. All of them. At once. On purpose, sort of. Let me tell you about it over coffee.
\nBeszel agents can talk to the hub in two directions, and the whole migration hinges on which one each agent uses.
\nIn push mode the agent dials out to the hub over a WebSocket. The agent is the one making the connection. Put the hub anywhere with a public address and the agent will find it, no inbound holes required.
\nIn pull mode the hub reaches in to the agent over an SSH-style channel on port 45876. The hub is the one making the connection, so it needs a route to the agent.
\nThat distinction is the difference between \"trivial migration\" and \"please open a hole from a cloud box into your home network,\" which is a sentence that should make anyone uneasy. If every agent pushes, the cloud hub never needs to touch my LAN. If any agent gets pulled, I would have to expose it. So before I moved a single thing, I needed to know exactly what I was dealing with.
\nThe hub keeps all of this in a little SQLite database. I pulled the list of systems out of it:
\nsqlite3 ~/beszel/data/data.db \\\n \"SELECT name, host, status FROM systems ORDER BY name;\"\nMost rows looked normal, a hostname and a LAN address. But a cluster of them listed a host of 192.168.1.1, an address the hub could not possibly reach, and yet their status was cheerfully up. That stopped me. How is a system reporting healthy if the hub cannot even route to the address it has on file for it?
The answer, once it clicked, is airtight. The host field means two completely different things depending on direction. In pull mode it is a destination the hub dials, so it has to be reachable. In push mode it is just the return address stamped on the connection when the agent phoned in, whatever the network happened to rewrite it to on the way. So an unreachable address plus an up status can only mean one thing: the hub is not reaching out to that box at all. The data is arriving on its own. That is a push agent, full stop, and no amount of squinting at the address changes it.
I verified the handful of ambiguous ones by actually reading each agent's config, but that one piece of logic did most of the work. When I finished, the tally was fifteen push and one pull. The one holdout was my container host.
\nThe container host was the only box the old hub was reaching in to. To move the hub to the cloud, that agent had to start pushing instead.
\nHere is the part I was nervous about. Beszel identifies a pushing agent by a fingerprint tied to a registration token. If I just flipped the agent to push, would it bind to the existing system record and keep all its history, or would it show up as a brand new duplicate and leave a ghost behind?
\nI went looking in the database and found that every system already has a row in a fingerprints table pairing it with a token. The pull agent's row had a token but an empty fingerprint, which is exactly the shape of \"a push agent has been provisioned here but has not connected yet.\" And the agent already held that same token in its config. So the moment it pushed, it would present the matching token, the hub would look it up, find the waiting row, and fill in the fingerprint. Same system, same history, no duplicate.
Which is precisely what happened. I added the hub URL to the agent, restarted it, watched the log say WebSocket connected, and checked the hub. Still one record. Fingerprint now populated. History intact. That is the good kind of boring.
The VPS is an OpenVZ container, and OpenVZ containers are fussy about the things Docker wants from a kernel. Rather than fight overlay filesystems and cgroup quirks, I ran the hub the way it ships, as a single Go binary under systemd. Fewer moving parts, nothing to argue with.
\n# /etc/systemd/system/beszel-hub.service\n[Unit]\nDescription=Beszel Hub\nAfter=network-online.target\nWants=network-online.target\n\n[Service]\nType=simple\nUser=beszel\nGroup=beszel\nWorkingDirectory=/opt/beszel\nExecStart=/opt/beszel/beszel serve --http 127.0.0.1:8090\nRestart=on-failure\nNoNewPrivileges=true\nProtectSystem=strict\nReadWritePaths=/opt/beszel/beszel_data\n\n[Install]\nWantedBy=multi-user.target\nNotice it binds to 127.0.0.1. The hub itself is never exposed. In front of it I put Caddy, which handles TLS and gets a real Let's Encrypt certificate automatically. The entire reverse proxy config is four lines:
beszel.example.com, metrics.example.com {\n reverse_proxy 127.0.0.1:8090\n}\nThat Let's Encrypt certificate turned out to matter more than I expected, and it quietly solved a problem I had been carrying. My old hub used a certificate from my private CA, which meant every agent had to trust my private root or it would refuse the connection and silently go offline. A public certificate is trusted by everything out of the box. Moving to the cloud with a real cert deleted an entire class of \"why is this agent down\" from my life.
\nTwo hostnames, on purpose. The UI lives at metrics.example.com and goes through Cloudflare's proxy, which is nice for a login page facing the open internet. The agents connect to beszel.example.com, which is a plain DNS record pointing straight at the box, no proxy. That split is not decoration. Beszel's agent WebSocket does not survive Cloudflare's proxy, it comes back with a 401 and the agent sulks. So the agents get a direct path and the humans get the proxied one, and everyone is happy.
I wanted the history, the alert rules, and my login to come across, so this was a database copy rather than a fresh start. Beszel uses SQLite, so a consistent snapshot is one command that does not even need the hub stopped:
\nsqlite3 ~/beszel/data/data.db \".backup '/tmp/data.db'\"\nI copied that to the VPS, dropped it into the new hub's data directory, and started it. Sixteen systems, all my history, my account. Then I upgraded the hub binary to the newest release and let it run its migrations against the copied database, which is the officially blessed upgrade path and went without a complaint. The new version had a feature I had been waiting on, container health alerts, so the upgrade was worth doing on its own.
\nOld hub still running at home the whole time, untouched, ready to flip back to. That detail becomes important in about two paragraphs.
\nHere is where I earned the coffee.
\nThe database I copied did not just bring my systems and history. It brought my alert rules and my notification config, which is to say my Telegram webhook. And the new hub, freshly started, looked at its sixteen systems, saw that almost none of them were talking to it yet because they were all still pushing to the old hub at home, and did exactly what I built it to do. It decided they were down. All of them. And it reached for the phone.
\nMy pocket started buzzing like an angry hornet. \"System down. System down. System down.\" Fourteen of them, in a tight little cluster, each one a small lie. Nothing was actually down. Everything was fine and happily reporting to the other hub. I had simply stood up a second watchtower, handed it a stale map, and it panicked on cue.
\nThe fix in the moment was to stop the new hub before it could send more. The lesson underneath it is the one worth keeping. When you run two monitoring hubs at once during a migration, and your agents are only reporting to one of them, the other hub sees a graveyard and starts screaming about it. And it is worse than that, because during a partial cutover the pain flips: as each agent moves to the new hub, the old hub loses sight of it and wants to alert too. Whichever hub an agent is not talking to will try to page you about it.
\nSo the real move, the one I should have made from the start, is to silence both hubs before you touch anything. I blanked the webhook on each one, saving the original first, so the rules could keep evaluating but had nowhere to send. Then I cut everything over in a batch, confirmed all sixteen were green on the new hub, and only then restored the webhook on the winner. Quiet migration, no hornets.
\nIf you take one thing from this post, that is it. A copied monitoring database is a loaded notification cannon. Point it at the wall before you plug it in.
\nWith both hubs muzzled, moving the agents was mechanical. For the Linux boxes the hub URL lives in the systemd unit, so it was a small edit and a restart, scripted across the fleet. The Mac agents keep it in an env file. The container host meant recreating a container. The TrueNAS box and the Home Assistant add on each needed a hand. One by one they lit up green on the new hub.
\nThen I noticed one box kept flickering. It would connect to the new hub, then a few minutes later drop and reappear on the old one. I would push it back, it would drift home again.
\nThe culprit was a fallback I had forgotten I built in. Every pushing agent also keeps that pull-style listener open on 45876 as a backup path, and the old hub still had all these systems in its list, so it was actively reaching in and grabbing them the instant a WebSocket so much as hiccuped. The old hub was not a passive has-been waiting to be retired. It was standing there with a fishing rod, reeling my agents back the moment my attention wandered.
\nThere is no clever fix for that. The old hub had to actually go. I stopped it and disabled it from ever starting again, and the flickering stopped instantly. The new hub was finally the only voice in the room.
\nI tune to my favorite TV Show on Bosnian TV via VPN because they only allow local broadcast. So here we are, last thing we need to do.
\nThe box now hosted a monitoring hub as well as a VPN, and it had no host firewall at all, just a wide open default policy leaning entirely on the fact that only a few things were listening. I wanted a proper default deny.
\nThe complication: as I sat down to do this, that same box was actively streaming my favorite show to a phone over WireGuard. Get this wrong and I do not just lock myself out, I cut the episode off right before the cliffhanger. That is not a mistake you get forgiven for.
\nSo the rules were surgical. Touch only the inbound chain, never the forwarding rules that carry the VPN traffic, put \"keep every connection that already exists\" as the very first rule so the live stream sails through untouched, and explicitly allow the VPN port on top of that for belt and suspenders. Allow SSH, the VPN, and the two web ports. Deny the rest.
\n# accepts first, while the policy is still open, so nothing drops mid-setup\niptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT\niptables -A INPUT -i lo -j ACCEPT\niptables -A INPUT -p icmp -j ACCEPT\niptables -A INPUT -p tcp --dport 49222 -j ACCEPT # ssh, on a non-default port\niptables -A INPUT -p udp --dport 51820 -j ACCEPT # wireguard\niptables -A INPUT -p tcp --dport 80 -j ACCEPT # acme\niptables -A INPUT -p tcp --dport 443 -j ACCEPT # caddy + beszel\n# and only now, the door\niptables -P INPUT DROP\nThe ordering is the whole trick. Every allow rule goes in while the policy is still open, so there is never a moment where a legitimate packet gets dropped because its rule had not been added yet. The drop policy goes on last, by which point the \"keep existing connections\" rule is already in place protecting both my SSH session and the stream.
\nAnd because I have locked myself out of exactly enough remote boxes to have learned, I did all of this with a second SSH session held open and a script already counting down to undo the whole thing in five minutes unless I confirmed it worked. I confirmed. The stream never so much as stuttered. I watched the VPN's byte counter climb straight through the change, which is the only proof I actually trust.
\nI reached for the modern firewall tool first, nft, and found it was not even installed on this little container, and its existing VPN rules were all in old-school iptables anyway. Mixing a fresh firewall framework into a box carrying a live stream is not a move you make on a whim, so I stayed on iptables to match what was already there. Right tool is sometimes just the one already in the room.
The hub works. It sits in a datacenter, watches my whole house from the outside, and if the house goes dark it is the one thing still standing to tell me. The paradox is resolved. The watchtower is finally outside the castle.
\nBut the running service is not really the souvenir. Three things are.
\nThe first is a piece of reasoning I will reuse forever: an unreachable address plus a healthy status can only mean the data is being pushed, not pulled. Systems tell you how they are wired if you read them carefully enough.
\nThe second is a scar. A copied monitoring database carries live notification config, and it will fire the instant it disagrees with reality. Silence your alerting before a migration, on every hub involved, and turn it back on only when the new world is actually true. I learned that at the cost of one buzzing pocket and a brief, sincere belief that my entire homelab had died.
\nThe third is the same reflex I keep relearning and keep being grateful for. Before you change anything that controls how you reach a box, or whether it can reach you, arm the undo first. The held session, the five minute timer, the saved-off original config. Every single time it has felt like overkill right up until the one time it was the only thing between me and a very long evening.
\nThe homelab still cannot tell me it is on fire from inside a fire. But now something outside can. That is the whole point, and it only took one fake apocalypse to get there.
\nI used this prompt to generate the featured image.
\nA cinematic tech-noir scene. A tall stone watchtower stands outside a castle wall, perched on a distant hill lit by warm amber light, while the castle itself sits in cool blue shadow across a dark valley. From the tower a single glowing beam of data, rendered as circuit-board traces and light, arcs across the valley toward the castle, watching over it. In the foreground a phone screen glows with a cascade of red \"system down\" alert bubbles, slightly out of focus, hinting at a false alarm. The tower is clearly safe and separate from whatever might happen to the castle. Deep blues and warm ambers, subtle 3D render mixed with clean vector line work, a faint terminal prompt glowing in one corner. A mood of calm vigilance earned through one chaotic night.
\n","comment_id":"6a9d7046ccab54000125377a","feature_image":"https://emir.fyi/content/images/2026/09/9c9c7523-e3e4-4fee-ab51-8513ba286bd2.png","featured":false,"visibility":"public","created_at":"2026-09-06T09:53:10.000-04:00","updated_at":"2026-09-07T10:18:36.000-04:00","published_at":"2026-09-07T10:18:36.000-04:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/my-monitoring-hub-lived-inside-the-thing-it-was-watching/","excerpt":"The Problem\n\n\nHere is a fun little paradox I let sit in my homelab for way too long. My monitoring hub, the piece of software whose entire job is to tell me when something in the house has fallen over, was running on a Mac Mini in the house. Same power. Same internet. Same failure domain as everything it was supposed to be watching.\n\n\nSo the day the power flickers or the ISP has a bad afternoon, here is what happens: the homelab goes dark, and the thing that is supposed to text me \"hey, the home","reading_time":12,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"6a987ef3ccab540001253767","uuid":"f6caf4fd-a6ae-4af3-8228-33cc332614d9","title":"My Router Kept Turning Itself Off, So I Bought a Real One","slug":"my-router-kept-turning-itself-off-so-i-bought-a-real-one","html":"Every few weeks, my entire house would lose the internet. Not the ISP. Not the WiFi. Everything, all at once, including things that have nothing to do with the internet. DNS stopped resolving, so none of my internal services could find each other. DHCP stopped handing out addresses, so anything that rebooted came up with no network at all. My phone would sit there spinning. My wife would ask, in a tone I have come to recognize, whether the internet was broken again.
\nThe culprit was a small fanless N100 mini PC I bought off AliExpress for about the price of a nice dinner. It ran pfSense CE perfectly well, right up until it didn't.
\nAnd the recovery ritual is what really got me. My rack is mounted on the wall, so bringing the network back meant climbing a ladder, carrying an HDMI cable, a wireless keyboard and a display up with me, plugging all of it into a firewall, and restarting the thing. At which point it would come up perfectly happily, as if nothing had ever been wrong, and I would climb back down and unplug it all again.
\nHere is the part that made it maddening. Just pressing reset did not work. Reset with a monitor and a keyboard attached worked every time. It was as though the box only agreed to boot if somebody was watching.
\nI ran last reboot one evening to see how bad the pattern was. Thirteen unclean shutdowns. That is not a router. That is a coin flip with an ethernet port.
I never chased down the root cause, but writing this up I went looking, and it turns out to be an extremely well travelled road. The UEFI firmware on a lot of consumer mini PCs simply refuses to complete boot when it cannot detect a display. Netgate's own forum has been collecting these for years: 2.4.0 does not boot without Monitor, Can't boot without monitor, a Shuttle DH110 doing the same thing, and an HDMI console going dark when pfSense boots with the monitor off.
\nThere are real fixes, and they are cheap. An HDMI dummy plug is a couple of dollars and makes the firmware believe a monitor is present. Disabling the serial ports in BIOS clears it on some boards. So does enabling CSM, or flipping the UEFI OS type. Any one of those might well have ended my ladder trips for the cost of a coffee.
\nI did not do any of them, and I want to be honest about why. It was not that I could not find the fix. It was that I no longer wanted to keep patching around a machine I had stopped trusting. Every workaround would have been another thing propping up a box that had already cost me thirteen outages and an unknown number of ladder climbs, and I would still never really know why it went down in the first place.
\nWhat I wanted instead was equipment from the people who actually make pfSense, which reviews consistently said was good. I had resisted spending money on that for a long time, because a firewall is deeply boring and there is always something more fun to buy. But once I added up what the cheap one had actually cost me in time and irritation, the maths stopped being close. If the new one simply does not break, it will have paid for itself.
\nSo I bought a Netgate 4200.
\nHere is the arc, so you can see where this is going:
\nSix steps. It took few hours, and almost none of that time was spent on the parts I expected.
\nLet me be fair to the N100 box for a moment, because it was not a bad machine.
\nIntel N100, 16 GB of RAM, four 2.5 GbE ports, completely silent, drew almost no power. It ran pfSense CE 2.8.1 with zero packages installed, which turned out to matter enormously later. It routed my whole network for a long time without complaint. For the money it was genuinely good.
\nThe problem was never performance. The problem was that when it failed, I had no way to diagnose it and nobody to ask, so my only tool was a ladder.
\nWhat I actually wanted was boring. A box whose entire job is to be a firewall, built by people whose entire job is building firewalls, that I never have to look at.
\nBefore I plugged anything in, I had one question I could not find a clean answer to anywhere:
\nDo I update the new appliance first, or restore my config first?
\nThis sounds trivial. It is not, and getting it backwards can waste an afternoon.
\nThe rule, once you find it buried in Netgate's documentation, is about the configuration revision, not the version number. Every pfSense release stamps a revision into config.xml. My CE 2.8.1 box was on revision 24.0.
And the rule is one directional:
\n| Direction | \nSupported? | \n
|---|---|
| Older config onto a newer release | \nYes, and it gets upgraded automatically | \n
| Newer config onto an older release | \nNo, rejected outright | \n
So if my new appliance had shipped with something older than my config's revision, restoring first would have simply failed. Update first, and the direction is always safe.
\nThere is a second ordering rule I have never seen written down anywhere, and it is arguably more important:
\nThe restore has to be the very last thing you do before cutover.
\nThe instant that config lands, the new box believes it is your gateway. It claims your gateway IP, starts a DHCP server, and starts answering DNS for all your internal hostnames. If it is connected to your live network at that moment, you now have two devices fighting over the same address and two DHCP servers racing to answer every request. That is a genuinely bad afternoon, and it is entirely avoidable by unplugging one cable.
\nSo the sequence is: update, then unplug WAN, then restore, then cut over.
\nThis is the best practical tip in this entire post, so I am putting it in its own section.
\nMy LAN runs on 10.10.10.0/24. The Netgate ships with a factory default LAN of 192.168.1.1/24. Those do not overlap.
Which means I could plug the new firewall's WAN port into a switch port on my existing network, plug my laptop into the new firewall's LAN port, and run the whole thing as a lab behind my live firewall while my house carried on completely unaware. The new box got internet through double NAT, which is perfectly fine for pulling updates. My laptop sat on 192.168.1.x on one side and 10.10.10.x on the other, with no conflict.
[ISP] → [old firewall] → [switch] → [new firewall PORT1/WAN]\n ↓\n [new firewall PORT2/LAN] → [laptop]\nNobody's Netflix was harmed. I could take as long as I wanted.
\nHere is the opinionated version: this is the single strongest argument for replacing a firewall with a second physical box rather than reinstalling the one you have. There is no equivalent safety net when you are reformatting your only router. You are committed the moment you boot the installer. With two boxes, you can fumble around for hours and roll back with two cables.
\nCheck your subnets before you rely on this. If your LAN happens to be on 192.168.1.0/24 you will need to temporarily move the new box to something else first, which is a five minute detour and still worth it.
Here is the trap that would have cost me an hour if I had not read the docs first.
\nOn the Netgate 4200, the front panel port numbers run in the opposite direction to the underlying device names:
\n| Front label | \nDevice name | \n
|---|---|
| PORT1 WAN | \nigc3 | \n
| PORT2 LAN | \nigc2 | \n
| PORT3 | \nigc1 | \n
| PORT4 | \nigc0 | \n
My old box had WAN on igc0 and LAN on igc1.
Read those two things together and you will see the problem. Those device names exist on the new box. They are just physical ports 4 and 3. So restoring my config unmodified does not throw an error, does not prompt for anything, and does not warn you. It quietly wires your WAN to the port silkscreened PORT4 and then sits there passing no traffic while you check cables and question your life choices.
\nEven the MAC addresses run backwards:
\nPORT1 / igc3 00:00:5e:00:53:a0 ← this is the MAC printed on the label\nPORT2 / igc2 00:00:5e:00:53:a1\nPORT3 / igc1 00:00:5e:00:53:a2\nPORT4 / igc0 00:00:5e:00:53:a3\nThe fix is trivial once you know. My entire 31 KB config file contained exactly two hardware specific strings, on lines 70 and 83. Everything else refers to wan and lan as logical names that do not care what silicon they land on.
# Remap WAN igc0 → igc3 and LAN igc1 → igc2 before restoring\nsed -i '' 's|<if>igc0</if>|<if>igc3</if>|; s|<if>igc1</if>|<if>igc2</if>|' config-remapped.xml\n\n# Always confirm you did not corrupt the XML\nxmllint --noout config-remapped.xml\nTwo lines. That is the entire hardware migration. It is worth pausing on how good that is: a firewall config with 36 DHCP reservations, 34 DNS overrides, NAT rules and a dynamic DNS client moved between completely different hardware by changing two strings.
\nThe reason it was that easy is that I had zero packages installed. No pfBlockerNG, no HAProxy, nothing. Packages are where migrations get ugly, because they carry their own state and their own version compatibility. If you are planning a hardware swap someday, the cheapest thing you can do today is resist installing packages you do not truly need.
\nI had a plan. The plan was good. Let me tell you about the four hours in the middle.
\nI unboxed it, plugged in PORT1 to my switch and PORT2 to my laptop, powered it on, and waited. The status LED did its blue flashing thing, which the documentation says means \"OS boot in progress.\" Then it stopped flashing.
\nMy laptop got a self assigned 169.254.x.x address. Nothing was serving DHCP. I could not reach the web interface. The old firewall saw no sign of the new box on the network at all, not a single frame.
Both ethernet ports had link lights. This turned out to be the most misleading fact of the entire day.
\nEthernet link lights come from the PHY, which powers up with the hardware and negotiates link whether or not the operating system has booted. A completely wedged box with a dead OS will still light up both ports and negotiate gigabit. I confirmed this later by accident: with the appliance fully shut down into standby, my laptop still reported status: active, 1000baseT <full-duplex> on that cable. Link lights tell you the hardware has power. They tell you nothing else. I will never trust them again.
I spent too long chasing whether the cable was bad, whether the switch port was dead, whether I had the wrong port. All of it was guesswork.
\nHere is the test that ended the guessing, and I recommend it any time you are stuck between \"the network path is broken\" and \"the service is broken\":
\n# Same source address, same destination, different ports\nnc -z -v -s 192.168.1.100 192.168.1.1 53 # Connection succeeded\nnc -z -v -s 192.168.1.100 192.168.1.1 443 # Operation timed out\nnc -z -v -s 192.168.1.100 192.168.1.1 80 # Operation timed out\nA full TCP handshake completed on port 53. A real DNS query came back in 0 msec. So the cable was fine, the switch was fine, the host was alive, and TCP worked. Ports 80 and 443 specifically had nothing listening.
\nThat one paired result replaced about ninety minutes of theorizing. When you cannot tell whether the path or the service is broken, find a port on the same host that does work. If anything answers, the path is proven and you can stop looking at cables.
\nThe diagnosis: the box had booted, Kea DHCP was running, unbound DNS was running, and the web server had simply never started. On a factory fresh appliance, out of the box.
\nA power cycle. Short press the power button, wait for the LED to go to pulsing orange, press it again. Ten minutes later the web interface answered.
\nI checked afterward whether slow certificate generation was to blame, since that can hold up the web server on a low power CPU. It was not. The GUI certificate had been issued at the factory weeks earlier. That first boot was just genuinely stuck.
\nI do not have a satisfying explanation, and I am suspicious of write ups that manufacture one. Sometimes brand new hardware needs to be turned off and on again, and the honest lesson is that a factory fresh appliance failing its first boot is a thing that can happen to you too.
\nThis bit me three separate times, so it is worth its own warning.
\nWhen the DHCP server appears after macOS has already given up and self assigned a 169.254.x.x address, macOS does not promptly try again. It sits there, contentedly wrong, for a long time. The box being fixed does not reach back and poke your laptop.
Reseating the USB-C end of my ethernet adapter fixed it every time. sudo ipconfig set en7 DHCP does the same thing without the physical fiddling.
The nastier version of this problem: while your interface is on 169.254.x.x, it has no route to the subnet you are trying to reach. So every ping, curl and port check you run silently falls through to your default route and goes out over WiFi instead. I did exactly this and briefly concluded the new firewall was unreachable when in fact I had never sent it a single packet. If you take one thing from this post, make it this: verify which interface your test actually used before you believe its result.
route -n get 192.168.1.1 # says which interface will be used\narp -an | grep 192.168.1.1 # says which MAC answered, and on which interface\nping -b en7 192.168.1.1 # forces the test onto a specific interface\nThis is my favourite failure of the day, because the evidence was so clean.
\nAfter the restore, I could not get an address on PORT2 where LAN was supposed to be. I moved the cable to PORT1, nothing. Moved it to PORT3, and immediately got a proper lease with the right gateway and the right DNS server.
\nI logged in and looked at the interface assignments:
\nWAN → igc0\nLAN → igc1\nThose are my old box's values. Which meant I had uploaded the original backup instead of the remapped one. The two files were sitting next to each other in the same directory with identical timestamps, and I grabbed the wrong one in the file dialog.
\nAnd it explains the symptom perfectly. LAN = igc1 = PORT3, which is exactly the port that gave me a lease. The mistake announced itself.
The fix did not need another restore. Interface assignments are two dropdowns in the web interface:
\nigc0 to igc3, save. Nothing breaks, since WAN has no cable in it.igc1 to igc2, save. Your connection dies immediately, which is success rather than failure. The change applies server side; the reply just cannot reach you because LAN is no longer on the port you are plugged into.Do those as two separate saves rather than one. There is no reason to lose connectivity before you have to.
\nBy this point the new box was sitting on my bench with the full config, correct interface assignments, and no WAN cable. My house was still happily running on the old flaky box, which had the decency not to switch itself off during any of this.
\nThe one thing I got right without being told: I had to physically move the appliance into the rack, which meant powering it down and unplugging it. My first instinct was to shut down, move it, cable everything up, and power on. That would have meant sitting through a ten minute boot with my whole house offline, hoping it came up correctly this time.
\nInstead:
\nThe slow, uncertain part happened while I was still safely online. Actual downtime was the cable swap plus waiting for a DHCP lease from my ISP. A couple of minutes.
\nPower cycle the modem. Your WAN MAC address just changed, and many ISPs bind the lease to it. Mine handed out an address after a modem restart with no further persuasion. If yours does not, pfSense lets you spoof the old MAC under Interfaces then WAN, which is why you should write the old one down before you unplug the box you can no longer read it from.
\nMy public IP changed during the cutover, from one address to a completely different one. This is exactly why I had recorded the old one before starting. Without that baseline I would have spent an hour wondering whether a changed public IP was a symptom of something I had broken.
\nThe Cloudflare dynamic DNS client picked it up on its own and updated my VPN hostname within a few minutes. There were some alarming errors in the log during the cutover window:
\nERROR [phpDynDNS] (vpn.example.com) Could not determine the request IP address\n (using \"wan\", \"igc3\"): gateway not online\nThose were transient. The WAN gateway simply had not come up yet. Once it did, the update went through. Worth knowing so you do not panic at your logs the way I briefly did.
\nEverything else came across intact:
\n| Check | \nResult | \n
|---|---|
| WAN and LAN | \nBoth negotiated 2500Base-T, correct addresses | \n
| DNS | \nEvery internal hostname resolving, recursion working | \n
| DHCP | \nAll 36 reservations honoured | \n
| NAT | \nWireGuard forward rebound to the new public IP automatically | \n
| Kubernetes | \nAll three nodes reachable through the API VIP | \n
| Internal services | \nEvery one of them answering | \n
| WireGuard from cellular | \nWorking | \n
That last one is the test worth doing deliberately. Connecting over WireGuard from your phone on mobile data exercises the NAT port forward and the dynamic DNS update at the same time, and it is the only check you cannot fake from inside your own network. If your DDNS had silently failed, that is where you would find out, ideally before you are in an airport rather than after.
\nOne false alarm worth mentioning, because it nearly sent me chasing nothing: my first sweep reported Home Assistant unreachable. Home Assistant listens on 8123, not 443. I had probed the wrong port. Through the load balancer it returned a perfectly healthy 200. Check what port a service actually listens on before you declare it broken.
\nLearn the config revision rule. Older config onto newer release works and gets upgraded. Newer onto older is rejected. Update the appliance first and you are always going the safe direction.
\nRestore last, cut over immediately after. The moment that config lands, the new box thinks it owns your network. Do not let it touch the live wire until the old one is off.
\nBench the new box behind the old one. If your subnets do not overlap, this costs nothing and buys unlimited time. It is the best reason to migrate onto a second box rather than reinstall your only one.
\nEthernet link lights mean nothing. They tell you the hardware has power. Not that the OS booted, not that anything is listening.
\nFind a port that works before you blame the cable. One successful connection on any port proves the entire path and saves you from an hour of cable swapping.
\nCheck which interface your test used. A self assigned address means no route, which means your test silently went out a different interface and lied to you.
\nWrite down the old WAN MAC before you unplug anything. You cannot read it off a box you have already disconnected, and you may need it if your ISP is fussy.
\nKeep the old box as a cold spare. Mine is sitting powered off with its config intact. Rollback is two cables and a button. That safety net is why the cutover was relaxed instead of terrifying.
\nResist installing packages. My migration was two lines of sed because there was nothing else to carry.
The part I keep coming back to is that almost nothing that went wrong was the thing I had prepared for. I had researched the port mapping trap carefully and it never bit me, precisely because I had researched it. What actually cost me the afternoon was a brand new appliance failing its first boot, an operating system that would not re-request DHCP, and me grabbing the wrong file out of a folder.
\nWhich is roughly how all of these go. The known risks get handled because they are known. The time disappears into the things nobody thought to write down.
\nSo I wrote them down.
\nI used this prompt to generate the featured image.
\n\n\n","comment_id":"6a987ef3ccab540001253767","feature_image":"https://emir.fyi/content/images/2026/09/40c3bbcc-4d72-4444-b7b9-076039cdd33c.png","featured":false,"visibility":"public","created_at":"2026-09-02T15:54:27.000-04:00","updated_at":"2026-09-02T16:15:13.000-04:00","published_at":"2026-09-02T16:14:22.000-04:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/my-router-kept-turning-itself-off-so-i-bought-a-real-one/","excerpt":"The Problem\n\n\nEvery few weeks, my entire house would lose the internet. Not the ISP. Not the WiFi. Everything, all at once, including things that have nothing to do with the internet. DNS stopped resolving, so none of my internal services could find each other. DHCP stopped handing out addresses, so anything that rebooted came up with no network at all. My phone would sit there spinning. My wife would ask, in a tone I have come to recognize, whether the internet was broken again.\n\n\nThe culprit w","reading_time":14,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"6a8db746ccab540001253754","uuid":"37f38146-d8d7-4e10-b162-a6f21a337a3f","title":"The Patch Was Easy. The Password Reset Almost Locked Me Out","slug":"the-patch-was-easy-the-password-reset-almost-locked-me-out","html":"A rack mounted white security gateway appliance seated cleanly in a home server rack, four ethernet ports with two blue cables plugged into the leftmost ports, small blue status LEDs glowing. Beside it on the shelf sits a small unbranded generic mini PC, powered off, cables coiled, clearly retired. Shallow depth of field, moody low key homelab lighting, teal and amber color grade, photographic realism, no text, no logos, no visible branding.
\n
Five months ago I wrote My Blog Told Me It Was Vulnerable. It Was Right. It was about CVE-2026-26980, a 9.4 CVSS SQL injection in Ghost's Content API, and about the mildly humiliating discovery that my only publicly exposed service was also the only one I'd never bothered to put in a compose file. I fixed the CVE, codified the stack, wrote down a standard, and ended the post with a line about how this would never happen again.
\nReader, it happened again.
\nThis time the email came from Ghost directly. Subject line \"Critical Ghost security update,\" the kind of thing that arrives at 10am on a Tuesday and rearranges your afternoon. A batch of new advisories had landed, and I should update as soon as possible and consider resetting my authentication credentials.
\nHere's the thing though. This time I was ready. The compose file existed. The standard existed. I knew exactly which host, which volume, which env file. The patch itself took about four minutes.
\nAnd then I nearly locked myself out of my own blog forever, because of a config value I'd never once thought about.
\nThe March incident had a certain clarity to it. One CVE, one number, one very large CVSS score. Nine point four is a number that makes you act. You read \"unauthenticated attackers can read arbitrary data from your database,\" you look at your publicly accessible blog, and you go patch it.
\nAugust was murkier. Seven advisories, published as a batch, and not one of them had a CVE ID assigned. Just GHSA identifiers and severity labels:
\n| Advisory | \nSeverity | \nPatched in | \nWas I exposed? | \n
|---|---|---|---|
| Editor oEmbed Preview Allows Untrusted Script Execution | \nHigh (8.1) | \n6.34.0 | \nYes | \n
| Input Validation Issue in Admin iframe Could Result in Staff Account Takeover | \nHigh (7.3) | \n6.34.0 | \nYes | \n
| Blind Password Hash Disclosure in Ghost Admin API | \nModerate (4.3) | \n6.58.0 | \nYes | \n
| Staff Sessions not fully Invalidated on Password Change | \nModerate | \n6.34.0 | \nYes | \n
| Staff Tokens Granted Elevated Post Privileges | \nModerate | \n6.58.0 | \nYes | \n
| Unauthenticated Comment Read in Private Mode | \nModerate | \n6.58.0 | \nYes | \n
| Denied Extension Bypass in Theme Serving via URL Encoding | \nModerate | \n6.20.0 | \nNo | \n
I was on 6.21.2. The highest patch floor in that list was 6.58.0, latest release was 6.59.0, so I was thirty eight minor versions behind and exposed to six of the seven. The theme serving bypass was patched in 6.20.0 and I'd scraped past it by two releases, entirely by accident. I'd like to claim that as good hygiene but it's just what happens when you patch once and then stop paying attention for five months.
\nI want to flag something about the psychology here, because I think it's the actually interesting part. A single 9.4 with a CVE number attached is easy to take seriously. Seven moderates and a couple of highs with no CVE IDs is much easier to file under \"I'll get to it.\" The March vulnerability was scarier. The August batch was more likely to be ignored. Those are not the same axis, and I only noticed because I happened to be looking.
\nThat fifth item is the post.
\nI want to be honest about how boring this was, because boring is the whole payoff from March.
\nBack up first. The content volume is 186MB of themes and images, the database dumps to about 400KB gzipped:
\n# mysqldump straight out of the running container\ndocker exec mysql-db sh -c 'MYSQL_PWD=$MYSQL_ROOT_PASSWORD mysqldump -uroot \\\n --single-transaction --routines --triggers ghost' | gzip > ghost-db.sql.gz\n\n# tar the content volume via a throwaway alpine container\ndocker run --rm -v ghost-content:/src:ro -v $PWD:/dst alpine \\\n tar czf /dst/ghost-content.tar.gz -C /src .\n\n# then actually verify them, because an unverified backup is a wish\ngzip -t ghost-db.sql.gz && tar tzf ghost-content.tar.gz > /dev/null\nThat last line matters more than it looks. A backup you haven't integrity-checked is a backup you're hoping about. It costs two seconds to know.
\nThen the upgrade, which was a one line diff:
\n services:\n ghost:\n- image: ghost:6.21.2\n+ image: ghost:6.59.0\nscp the compose file up, docker compose up -d, wait. Ghost ran its migrations for 107 seconds, chattering about renaming columns and adding indexes and one slightly ominous line about gift_links that turned out to be fine. Then:
[INFO] Database is in a ready state.\n[INFO] Ghost booted in 109.857s\nTwenty three posts intact. Site up. Admin up. That was the whole upgrade. In March this same operation was an evening project because I had to reverse engineer my own container first. This time it was a text edit and a file copy. The standard held, which is a genuinely nice feeling and also the last nice feeling in this post for a while.
\nI rotated the MySQL credentials the same way as last time, and I'll repeat the gotcha from the March post because it's still the thing that bites people: MYSQL_ROOT_PASSWORD and MYSQL_PASSWORD only do anything on first initialization of the data directory. Changing them in your .env does not change the password in the database. You ALTER USER inside MySQL first, then update the env file to match, then recreate. Other order, and your fresh container confidently presents a password the database has never heard of.
Ghost's advice was to update and consider resetting authentication credentials. There's an official flow for this, and it's genuinely well built. Settings, Advanced, Danger zone, \"Reset all authentication.\" It rotates every API key, signs out every staff user, and forces a password reset. Exactly what you want after a batch of advisories that includes password hash disclosure and incomplete session invalidation.
\nI had my cursor over the button. And then, for no particularly noble reason, I decided to check how Ghost was configured to send email.
\ndocker exec emir.fyi-blog sh -c 'cat /var/lib/ghost/config.production.json'\n{\n \"mail\": {\n \"transport\": \"Direct\"\n }\n}\nDirect means Ghost attempts to deliver mail itself, connecting straight to the recipient's mail server from wherever it happens to be running. Where it happens to be running, in my case, is a Docker host on a residential connection behind a Cloudflare tunnel. Gmail's opinion of unauthenticated mail arriving directly from a consumer IP block is not a warm one. That mail was going nowhere.
Now put those two facts next to each other.
\n\"Reset all authentication\" signs you out and requires a password reset. The password reset arrives by email. I have exactly one staff account. If I had clicked that button ten seconds earlier, I would have been signed out of a blog I could not sign back into, holding a reset email that was never going to be delivered.
\nRecoverable? Sure. I have database access, and a bcrypt hash written directly into the users table would have gotten me back in eventually. But \"eventually,\" on a Tuesday afternoon, via manual surgery on my own auth table, is not how I wanted to spend the day. And it would have been entirely self-inflicted.
The worse realization came a minute later. Ghost 6 has Device Verification, which emails you a six digit code when you sign in from a new device. That feature had been silently useless on my install for its entire existence. I'd simply never signed in from a new enough device to find out. The blog had a broken dependency sitting quietly in the middle of its authentication flow, and the only thing keeping it invisible was that I'd never needed it.
\nThis is the failure mode I actually want to write down. It was not a bug. Nothing was broken, crashed, or alerting. Every dashboard was green, the site served traffic all day, and Beszel was perfectly happy. It was a capability I had never exercised, which meant it had never had a chance to fail where I could see it. My monitoring watched whether Ghost was up. Nothing on earth was watching whether Ghost could send an email.
\nI already run Brevo SMTP for other services in the homelab, so the fix was mostly plumbing. Ghost takes config as env vars with double underscores for nesting:
\nmail__transport=SMTP\nmail__options__host=smtp-relay.brevo.com\nmail__options__port=587\nmail__options__secure=false\nmail__options__auth__user=your-smtp-user\nmail__options__auth__pass=your-smtp-password\nmail__from=Your Blog <ghost@example.com>\nTwo notes. Set secure: false for port 587, which uses STARTTLS. true is for 465. And the from address has to be on a domain you've actually authenticated in your relay. My blog domain isn't verified in Brevo but another domain I own is, so mail goes out from there. Slightly inelegant, completely functional.
Then I verified it, in three escalating steps, because I'd just been reminded what assumptions are worth.
\nStep one, does the config exist. Trivially checkable and almost meaningless, but it rules out typos:
\ndocker exec emir.fyi-blog env | grep '^mail__'\nStep two, does the relay accept us. Ghost bundles nodemailer, so you can borrow it to test the handshake without sending anything:
\ndocker exec emir.fyi-blog node -e \"\nconst nm = require('/var/lib/ghost/current/node_modules/nodemailer');\nnm.createTransport({\n host: process.env.mail__options__host,\n port: Number(process.env.mail__options__port),\n secure: false,\n auth: { user: process.env.mail__options__auth__user,\n pass: process.env.mail__options__auth__pass }\n}).verify()\n .then(() => console.log('SMTP AUTH: OK'))\n .catch(e => console.log('SMTP AUTH: FAILED', e.message));\n\"\nSMTP AUTH: OK\nStep three, does a human receive it. This is the only step that counts, and I want to be emphatic about why. I have a note in my TODO from a previous adventure with a different provider that shows 100 sent, 100 delivered, and not a single email ever arriving in the inbox. Not in spam. Not in promotions. Just gone. \"The relay accepted it\" and \"a person read it\" are separated by SPF, DKIM, DMARC, reputation scoring, and the inscrutable moods of large mail providers.
\nSo I triggered a real Ghost password reset, through Ghost's own code path, and went and looked at an actual inbox:
\ncurl -X POST http://localhost:2368/ghost/api/admin/authentication/password_reset \\\n -H 'Content-Type: application/json' \\\n -H 'X-Forwarded-Proto: https' \\\n -d '{\"password_reset\":[{\"email\":\"you@example.com\"}]}'\nIt arrived. Better still, signing back in triggered a Device Verification code, which also arrived, confirming that a feature I'd never successfully used in my life was now working.
\nThen I clicked the Danger zone button.
\nWorth documenting, since the docs describe it in general terms and I had the database open anyway:
\nAPI keys rotated in the last 10 minutes: 9\nActive sessions: 5 → 1\nPosts: 23 (unchanged)\nStaff account: active\nThe sign out was immediate and total. I landed on a page reading \"Update your password. For security, you need to create a new password. An email has been sent to you with instructions.\"
\nThat sentence is the whole post in miniature. Twenty minutes earlier, that page would have been a wall.
\nOne consequence I did see coming. I proxy a public JSON feed of my posts at /api/posts through Caddy, which injects a Ghost Content API key server side so callers don't have to:
@api_posts path /api/posts /api/posts/\nhandle @api_posts {\n handle {\n rewrite * /ghost/api/content/posts/?key={{ ghost_content_api_key }}&limit=all&{http.request.uri.query}\n reverse_proxy 192.168.1.10:2368 {\n header_up X-Forwarded-Proto https\n }\n }\n}\n\"Rotate every API key\" includes that one. The moment the reset completed, the feed started returning 401. Entirely expected, briefly satisfying to watch, and then I had to go fix it.
\nThe fix itself was unglamorous. That key is templated into the Caddy config on my load balancer nodes, so it got the new value and Caddy got a reload. Ten minutes, no drama. If you inject an API key anywhere in your infrastructure, write down where, because \"rotate all keys\" is a bigger button than it looks and the reverse proxy will not tell you it's now serving a dead credential.
\nThe part worth keeping is how I checked it afterward. Verify every branch, not just the one you were thinking about:
\ncurl -o /dev/null -w \"%{http_code}\\n\" \"https://emir.fyi/api/posts\" # 200\ncurl -o /dev/null -w \"%{http_code}\\n\" \"https://emir.fyi/api/posts?limit=1\" # 200\ncurl -o /dev/null -w \"%{http_code}\\n\" \"https://emir.fyi/api/posts?filter=slug:x\" # 200\nMy config has separate handlers for \"caller passed a limit\" and \"caller didn't.\" Testing only the default path would have left the other one quietly broken with the old key, and I'd have found out from a stranger.
\nMarch's lesson was about deployment. I had a service with no compose file, and the CVE forced me to codify it. I ended that post pleased with myself for establishing a standard.
\nThe standard worked. That part's real. The patch was a one line diff and a file copy, exactly as designed.
\nBut codifying deployment only answers \"how do I change this thing.\" It says nothing about \"can I recover this thing.\" Those feel adjacent and they are not. Every piece of my March work was about the deploy path, and the thing that nearly bit me was on the recovery path, which I had never walked, never tested, and never thought about.
\nThe specific shape of it is worth internalizing. A broken deploy is loud. Containers crash loop, healthchecks go red, monitoring pages you. A broken recovery path is completely silent, because by definition you aren't using it. Mine had been broken for the entire life of the install and the only way I found out was by hesitating over a button for no reason.
\nSo the question I'm adding to my list, for every service I run: if I locked myself out of this right now, what's the path back in, and when did I last confirm that path works? For Ghost the answer was email, and the answer to the second half was \"never.\"
\nThe blog is on 6.59.0 with fresh everything. But the actual repair today wasn't the version bump. It was noticing that the fire escape had been welded shut since the day I moved in.
\nI used this prompt to generate the featured image.
\n\n\n","comment_id":"6a8db746ccab540001253754","feature_image":"https://emir.fyi/content/images/2026/08/cc3b31d4-fea4-4e77-a696-e5fb946a43dc.png","featured":false,"visibility":"public","created_at":"2026-08-25T11:39:50.000-04:00","updated_at":"2026-08-25T12:08:27.000-04:00","published_at":"2026-08-25T12:08:27.000-04:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/the-patch-was-easy-the-password-reset-almost-locked-me-out/","excerpt":"The Patch Was Easy. The Password Reset Almost Locked Me Out.\n\n\nFive months ago I wrote My Blog Told Me It Was Vulnerable. It Was Right. It was about CVE-2026-26980, a 9.4 CVSS SQL injection in Ghost's Content API, and about the mildly humiliating discovery that my only publicly exposed service was also the only one I'd never bothered to put in a compose file. I fixed the CVE, codified the stack, wrote down a standard, and ended the post with a line about how this would never happen again.\n\n\nRead","reading_time":10,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"6a10832ea2778d0001eecb22","uuid":"6128b7b7-5959-46a0-b137-587dcd6c87ba","title":"Cooling My Garage Servers While I'm at the World Cup","slug":"cooling-my-garage-servers-while-im-at-the-world-cup","html":"A dimly lit server room at night, a single rack glowing with soft blue status LEDs, all indicators green and healthy. In the foreground, a heavy steel emergency exit door stands slightly ajar with warm light spilling through it, but the door frame is crudely welded shut with thick industrial weld beads along its edge. A small brass envelope icon hangs from the door handle like a key tag. Cinematic lighting, shallow depth of field, moody teal and amber color grading, photorealistic, 16:9 aspect ratio.
\n
A few of my homelab boxes live in the garage. In Florida. The garage stays manageable most of the year, the rack vents to ambient, and the Everwell mini split bolted to the wall keeps things comfortable whenever I remember to turn it on. The \"remember\" part is the load-bearing word in that sentence. It's also my office where I work from so I \"remember\" when it gets hot lol
\nThe 2026 World Cup is happening here. Some of it close enough that I'm not going to be the guy with three host cities on his doorstep and stay home watching it on a laptop. So I'm going. For chunks of weeks at a time. To matches. With my servers sitting in a room that can hit triple digits in June if the AC is off long enough.
\nNone of those servers care about being warm, until they suddenly do, in the form of thermal throttling, then noisy fans, then the kind of silent permanent damage that you only discover three months later when a drive starts failing SMART checks. I'd rather not come back from a group stage match to a brick.
\nThe Everwell only listens to its IR remote, which is sitting on my desk. So I needed something to act as a substitute for me. A small, dumb hand that holds the remote and can be told what to press by Home Assistant. That ended up being a BroadLink RM4 mini, plus a slightly larger pile of yak shaving than I expected.
\nPhased so future me can see what's done and what's pending.
\nFive of six done. The sixth one is just time.
\nThe shortlist was small. I needed something cheap, IR only, with a real Home Assistant integration, and not so weird that it would require me to read someone's GitHub gist from 2021 to make it work.
\n| Device | \nVerdict | \n
|---|---|
| BroadLink RM4 mini ($25) | \nWhat I picked. IR only, mature HA integration, well-documented | \n
| BroadLink RM4 Pro ($45) | \nRF + IR. The RF half is wasted budget for an IR-only mini-split | \n
| SwitchBot Hub Mini | \nConcept's the same but you're more married to their app | \n
| Sonoff IR blaster | \nCheaper, flakier reputation, dimmer HA community support | \n
| \"Just buy a smart AC\" | \nNot happening. My skills don't go beyond this split ac in terms of being an AC tech, I installed it myself and it would have to work | \n
The RM4 mini is a small black puck about the size of a film canister. USB-powered, 2.4 GHz Wi-Fi only, omnidirectional IR LED, plus a free bonus temperature + humidity sensor that you get whether you want it or not. (You want it. More on that later.)
\nThe pairing dance was the most \"consumer IoT\" part of the build. Three things to know up front:
\nThe actual pairing was about 90 seconds. Plug the RM4 in, the LED on the front rapid-blinks at roughly 2 Hz, the app discovers it on the LAN, you hand it your Wi-Fi password, done. The app immediately starts showing the device's onboard temp and humidity sensor, which felt magical for a $25 puck.
\nThe Everwell remote is the \"Full State Display\" kind, the better of the two flavors. Snowflake icon, fan-speed bars, setpoint digits, all on the LCD. Every IR transmission encodes the entire state (mode, temp, fan, on/off) in one burst, which means discrete on/off codes exist. No toggle ambiguity, no \"did I just turn the AC on twice.\"
\nSo I went down the obvious path first. BroadLink app -> Add Appliance-> AC Remote Full Display -> let the app pattern-match against its codeset library.
\nThis is where it got annoying. The \"User Defined Panel\" flow walks you through:
\nUnder the hood it's guessing your AC's protocol from a library of known codesets and trying them in sequence. The hope is one matches and the test press turns your AC back on.
\nFor Everwell, no dice. The test sends a signal that looks plausible to the app but doesn't speak my AC's specific dialect. Weirdly, it would recognize the off code but not the on. Off brand mini splits are a graveyard of \"close but wrong\" rebadges (Hisense, TCL, Midea, etc., all OEMing the same physical units with subtly different IR encodings).
\nThe fix: stop pattern matching, start recording.
\nIn the BroadLink app the option is buried at the bottom of the \"Add Appliance\" grid, called UserDefine. It makes an empty virtual remote with no semantics, just blank buttons. You add a button, hold the original remote a few inches from the RM4, press the physical button you want to clone, and the RM4 captures the raw IR burst byte for byte. Replay later equals the same bytes back out. Works on anything that emits IR.
\nTwo captures were all I needed:
\nOne side effect to expect. At three inches the IR scatters in every direction, including across the room to the AC. So you ll often see the AC actually respond mid capture (turn on or off). I leaned into it: each capture's scatter happens to leave the AC in exactly the right state for the next capture (off → on → ready for off → off). Convenient.
\nThe takeaway I wish someone had handed me before I spent twenty minutes in the state matcher: for off brand mini-splits, skip the codeset library and go straight to UserDefine. It's slower if you want every button on the remote, but you only ever need two or three buttons for \"control AC from a stadium parking lot\" use cases.
\nIn the world of consumer IoT, this is supposed to be the easy part. Settings -> Devices & Services -> Add Integration -> Broadlink -> enter IP -> done.
\nIt got to \"enter IP,\" then died with Invalid authentication. The full error explained itself: the RM4 was locked to the BroadLink cloud account I'd paired it with. Anything not signed in as that account, including my own LAN's Home Assistant, gets rejected. The mitigation is right there in the dialog: open the BroadLink app, tap the device, tap the three-dot menu, scroll to the bottom, disable the lock. Once disabled, HA authenticates immediately and the integration adds cleanly.
Now the fun part. The IR codes I'd captured back in Phase 2 live in BroadLink's cloud, not on the device itself, and BroadLink does not give you any way to export them. So in HA I had to re learn the same two codes, this time directly into HA's own storage.
\nDeveloper Tools → Actions → remote.learn_command:
action: remote.learn_command\ntarget:\n entity_id: remote.ac_garage\ndata:\n command_type: ir\n device: Garage AC\n command: power_on_74f\nClick \"Perform action,\" RM4 LED starts blinking (learn mode, 30-second window), point the Everwell remote at it, press Power, green toast. Repeat with command: power_off. Two minutes, both codes captured.
The relearn is a feature, not a chore. After this, the only thing in the loop is local hardware: HA -> LAN -> RM4 -> IR LED -> AC. The BroadLink cloud could vanish tomorrow and nothing about my AC control changes. That was the whole point.
\nThe integration also auto-exposes the RM4's onboard temperature and humidity sensors as sensor.ac_garage_temperature and sensor.ac_garage_humidity. Free thermometer. Reads about 1-2°F warm because of self heat from the RM4's own electronics, but plenty accurate for \"is the garage cooking\" decisions.
Two captured codes plus a temperature sensor is enough to close the loop. The automation I actually want:
\n\n\nWhen the garage gets too hot for too long, blast the AC for a while, then turn it off, then wait long enough that I'm not bouncing the compressor every other minute.
\n
Translated into HA's automation YAML, in /config/automations.yaml:
- id: garage_ac_burst_cool\n alias: Garage AC burst cool when hot\n description: Burst cool the garage when temperature spikes\n trigger:\n - platform: numeric_state\n entity_id: sensor.ac_garage_temperature\n above: 92\n for:\n minutes: 5\n condition:\n - condition: time\n after: \"09:00:00\"\n before: \"19:00:00\"\n action:\n - service: remote.send_command\n target:\n entity_id: remote.ac_garage\n data:\n device: Garage AC\n command: power_on_74f\n - delay:\n minutes: 10\n - service: remote.send_command\n target:\n entity_id: remote.ac_garage\n data:\n device: Garage AC\n command: power_off\n - delay:\n minutes: 10\n mode: single\nA few moving parts worth pointing at:
\nfor: minutes: 5 on the trigger. The sensor has to read above 92°F continuously for five minutes before this fires. A brief spike (someone opens the garage door for 20 seconds, the IR scatter from another capture, whatever) doesn't kick the AC.condition: time restricts firing to between 9am and 7pm. The garage barely gets above the threshold overnight, and I don't want the AC kicking on at 3am when the heat dump from the sun isn't even a factor yet.delay: 10 min is the cooldown. While the automation is mid execution, including that final delay, mode: single blocks new trigger fires. So even if the temperature stays above 92°F the whole time, the automation can't start a second burst until those final ten minutes elapse.I ran it manually first to verify the action chain works without waiting on the temperature trigger. The HA \"Traces\" view drew it as a clean timeline: trigger at T+0, send power_on_74f, wait, send power_off, wait, complete. The AC turned on, ran, turned off. Exactly as designed.
And then the next morning the automation fired for real, on a 92°F garage, and the AC didn't actually turn on. I'd lost a substitution before kickoff.
\nThis is the part of the post where I admit I almost shipped a broken thing.
\nThe trace said success. Trigger fired at 9:55:34 AM, condition passed, both remote.send_command calls completed without errors, all four steps finished. From HA's point of view, everything was great. From the garage's point of view, nothing happened.
The smoking gun was the temperature graph. During the supposed 5-minute \"on\" window (this was an earlier version of the YAML with a shorter burst), the sensor stayed flat at 92.4°F. No dip. Not even a wobble. If the AC had actually run, even briefly, the sensor sitting on the RM4 across the room would have seen it.
\nSo I tested the exact same command manually a few minutes later. AC turned on instantly. Sensor went 92.93 → 89.6°F over the next six minutes. The IR pipeline worked. The codes worked. Everything was fine. The 9:55 attempt had just silently lost the IR pulse somewhere.
\nThe hard lesson: remote.send_command returning success only means HA spoke to the RM4 over the LAN. It does not mean the RM4 actually emitted clean IR. And it definitely doesn't mean the AC heard it and acted on it. IR is fire and forget. There is no ack. There is no retry. If a UDP packet between HA and the RM4 drops, or the RM4 IR pulse arrives during a moment the AC isn't paying attention, the burst silently fails and the automation still thinks it succeeded.
The next automatic fire later that day went perfectly. Trigger at 11:53:34 → AC kicked in at 11:55:34 → sensor went 92.39 → 89.42°F over the next 18 minutes. Curve looked exactly like what a working AC should look like.
\nSo the pipeline works most of the time, not always. Which is to say: this is a real world IoT system, not a science experiment.
\nI haven't solved this yet. The roadmap of \"what would actually fix it\":
\npower_on, sleep 90 seconds, check if the sensor dropped at all. If not, re-send. Self-healing automation.For the World Cup trip I'll probably do (1) plus rest is a separate weekend project.
\nLast piece is being able to reach the dashboard from the airport, the stadium, the hotel, wherever. The good news: I already solved this problem a while back, just for the rest of the homelab instead of HA specifically. I wrote about that build in detail in WireGuard VPN to My Homelab: Automating a VPN Server with Terraform and Ansible. Short version: there's a small Ubuntu VM running WireGuard that terminates inbound UDP from anywhere on the internet (via a Cloudflare DDNS record plus a pfSense port-forward) and routes the client back onto my LAN. Phone, laptop, whatever, once the tunnel is up every internal service is reachable by its private hostname.
\nThe reasoning for WireGuard over public ingress is the usual two:
\nBecause everything's behind WireGuard, there is no new HA-specific work for remote access. The same https://homeassistant.localdomain URL I use on the couch works from a hotel Wi-Fi the moment the tunnel comes up. Same login, same dashboard, same buttons. The whole stack is identical on LAN vs over WG, which is the entire point of running the VPN as a subnet router instead of per-service tunnels.
Verified it before declaring victory: phone on cellular, Wi-Fi off, tunnel up, dashboard loads. Done.
\nA few things, ordered by how much they surprised me:
\nremote.send_command is not a verification, it's an aspiration. It tells you HA sent the packet to the RM4, not that the AC received the IR. For anything important, build verification into the automation, not faith.mode: single plus a trailing delay is the cleanest way to add a cooldown to an automation. No timers, no input booleans, no helper scripts. The automation stays \"running\" through the cooldown, and mode: single blocks new triggers for the duration. Simple.And one thing I didn't really learn so much as confirm: a $25 IR puck is genuinely all you need to bring a dumb mini-split into your homelab. The hardware is the cheap part. The software glue around it, and the part where you find out your \"successful\" automation didn't actually do anything, is where you spend the actual time.
\nI'm not 100% confident this rig will hold up across a full multi week absence. The first time it does, though, I'll write a follow up post titled \"I watched a World Cup match and my garage stayed at 78°F.\" Until then, I have plane tickets and a thermostat substitute on the bench. Let's see if the IR blaster can hold the line.
\nI used this prompt to generate the featured image.
\nA small black cylindrical IR blaster sitting on a workbench in a sun-lit Florida garage, with a partially-visible server rack glowing softly in the background. The blaster is emitting a stylized infrared beam toward an off-camera wall-mounted mini-split air conditioner. On the wall behind the bench, a 2026 FIFA World Cup wall calendar with a few match dates circled in red marker. Warm late-afternoon sunlight slicing through a half-open garage door, palm fronds just barely visible outside. Cinematic homelab aesthetic, shallow depth of field, photorealistic, subtle blueprint-overlay graphic in one corner showing the signal flow: phone (on the road) → WireGuard → Home Assistant → IR blaster → AC. Square 1:1 aspect ratio.
\n","comment_id":"6a10832ea2778d0001eecb22","feature_image":"https://emir.fyi/content/images/2026/05/b138110c-a75c-482d-b432-a34b90518333.png","featured":false,"visibility":"public","created_at":"2026-05-22T12:24:14.000-04:00","updated_at":"2026-05-22T12:41:41.000-04:00","published_at":"2026-05-22T12:41:41.000-04:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/cooling-my-garage-servers-while-im-at-the-world-cup/","excerpt":"The Itch\n\n\nA few of my homelab boxes live in the garage. In Florida. The garage stays manageable most of the year, the rack vents to ambient, and the Everwell mini split bolted to the wall keeps things comfortable whenever I remember to turn it on. The \"remember\" part is the load-bearing word in that sentence. It's also my office where I work from so I \"remember\" when it gets hot lol\n\n\nThe 2026 World Cup is happening here. Some of it close enough that I'm not going to be the guy with three host ","reading_time":12,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"69e41b613cad350001301cf9","uuid":"dfcf8ef9-75f3-4544-a9a9-55ec74f4ce75","title":"Building a High Availability Kubernetes Cluster Across Mixed Hardware (Part 2: Migration and the Graceful Goodbye)","slug":"building-a-high-availability-kubernetes-cluster-across-mixed-hardware-part-2-migration-and-the-graceful-goodbye","html":"Part 1 ended with a shiny new 3-node HA Kubernetes cluster: one Proxmox VM and two BeeLink mini PCs, glued together with kubeadm, kube-vip, and Flannel. The control plane was HA, the etcd quorum was solid, and the VIP survived any single-node failure.
\nThe cluster was also empty.
\nI closed that post with \"the hard part is done, the easy part is next, famous last words.\" Reader, I was not entirely wrong. Migrating the workloads was significantly easier than building the cluster. But \"easy\" is doing a lot of work in that sentence, and there were a few moments where I questioned my life choices.
\nThis is the story of moving three stateful services from a single-master cluster that was one bad RAM stick from oblivion, to a new HA cluster running on physically separate hardware. Without losing data. Without extended downtime. While running both clusters in parallel and keeping the option to roll back at every step.
\nvault.localdomain, argocd.localdomain, and market-mind.localdomain land on the new clusterThe thing that made this migration bearable: both clusters run at the same time, and the HA load balancer (Caddy) decides which one handles any given hostname. Flipping vault.localdomain from old to new is a one-line change in the Caddyfile and a playbook re-run. Flipping back is the same change in reverse.
Every migration step followed the same shape:
\nkubectl port-forward (no DNS change, no blast radius)This feels obvious in retrospect, but the previous me would have been tempted to do \"one big cutover at midnight\" and spend an hour panicking. Parallel clusters turn \"cutover\" into \"routing decision\" and routing decisions are cheap.
\nBefore any real workloads, I needed four things on the new cluster.
\nSame version as the old cluster (traefik-39.0.0, Traefik v3.6.7), installed via Helm. Then a small NodePort service to expose 80:30080 and 443:30443 so the HA LB's Caddy can reach it:
apiVersion: v1\nkind: Service\nmetadata:\n name: traefik-nodeport\n namespace: traefik\nspec:\n type: NodePort\n selector:\n app.kubernetes.io/name: traefik\n ports:\n - name: web\n port: 80\n targetPort: web\n nodePort: 30080\n - name: websecure\n port: 443\n targetPort: websecure\n nodePort: 30443\nTest: curl -sk https://192.168.1.40:30443/ on every cluster node. Got 404 on all three, which is exactly right. No IngressRoutes yet, so Traefik has nothing to route. 404 means \"I'm here, I just don't know that hostname\" and that's the healthy answer.
Rancher's local-path-provisioner v0.0.30, same version as the old cluster. Creates a local-path StorageClass that turns node-local disk into PVCs. Fast, simple, and the honest truth that homelab storage is usually pinned to wherever the pod first scheduled.
Tested with a throwaway PVC and pod that wrote a file, then read it back. PVC bound, file present, done. Next.
\nVaultwarden and MarketMind both use Bitnami Sealed Secrets so their secrets can live in git encrypted. The controller encrypts with a public key and decrypts with a private key that lives only in the cluster.
\nThe catch: if you install a fresh Sealed Secrets controller on a new cluster, it generates a new private key. Existing sealed secrets in your repo, encrypted against the old key, will never decrypt. They just sit there looking broken.
\nThe fix is to restore the old keys before installing the controller:
\n# 1. Export all sealed-secrets keys from the old cluster\nkubectl --kubeconfig ~/.kube/config-old \\\n get secret -n kube-system \\\n -l sealedsecrets.bitnami.com/sealed-secrets-key \\\n -o yaml > keys-backup.yaml\n\n# 2. Apply them to the new cluster BEFORE installing the controller\nkubectl --kubeconfig ~/.kube/config-ha apply -f keys-backup.yaml\n\n# 3. Now install the controller\nkubectl --kubeconfig ~/.kube/config-ha apply \\\n -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.27.3/controller.yaml\n\n# 4. Delete the backup file\nrm keys-backup.yaml\nThe controller starts up, sees the restored keys, registers them, and happily decrypts existing sealed secrets from the repo. Log output:
\nINFO msg=\"Searching for existing private keys\"\nINFO msg=\"registered private key\" secretname=sealed-secrets-key7m2wg\nINFO msg=\"registered private key\" secretname=sealed-secrets-keykzqw4\nINFO msg=\"registered private key\" secretname=sealed-secrets-keyt4fnz\nThree keys because the controller rotates them periodically and keeps old ones around for backward compatibility. All three now live on both clusters. Lesson: back up the Sealed Secrets private key the day you install the controller. Put it in your password manager. Without it, every sealed-secret.yaml in your repos becomes encrypted garbage.
The wildcard cert for *.localdomain was still valid for another year, so I just copied the wildcard-localdomain-tls Secret from the old cluster's vaultwarden namespace into the new cluster's vaultwarden, argocd, and later market-mind namespaces. Same cert works on both sides.
Vaultwarden was first because it's the highest-value service (password manager) and also a clean test case: single pod, single SQLite database, single PVC. If the pattern works here, it works everywhere.
\ntar czf /tmp/vw-data.tar.gz the /data directory (SQLite db, attachments, rsa_key.pem, icon cache)kubectl cp the tarball out to my Mackubectl cp the tarball in, tar xzf, doneTotal data size: 3.6MB. Total downtime on the new side: a few minutes. Old cluster data was frozen the entire time.
\nVaultwarden responded. HTTP 200. API config returned the right domain. Cert was valid. Everything on the technical side was green. So I flipped the Caddyfile:
\n vault.localdomain {\n tls /etc/caddy/certs/wildcard.crt /etc/caddy/certs/wildcard.key\n- import k8s_backend\n+ import k8s_backend_ha\n }\nReran the Ansible playbook, Caddy reloaded on all three LB nodes in a second, tested curl https://vault.localdomain/api/config and the right response came back. Felt great.
Then I tried to actually log in from my Bitwarden client.
\nWrong password.
\nI tried it again. Wrong password. A cold feeling started forming in my stomach. Did the SQLite WAL files not flush? Did the copy corrupt something? Was I about to lose years of passwords?
\nBefore changing anything, I asked my pair-assistant to flip the DNS back to the old cluster and scale the old Vaultwarden back up. Took about 30 seconds. Tried the same password on the old one.
\nWrong password.
\nThat's when I realized I'd been typing my email wrong. I had fat-fingered one character. The migration was fine. I was the bug.
\nWe flipped back to the new cluster, I logged in successfully, and I learned a lesson that applies to every migration: when something doesn't work, verify it doesn't work on the known-good side first. Don't assume the thing you just changed is the thing that broke. Sometimes you're just typing your own email wrong.
\nArgoCD is the thing that manages the other things. It watches my MarketMind repo, notices when I push a manifest change, and syncs it into the cluster. Migrating ArgoCD itself is basically \"install it again and re-register the same git repos.\"
\nInstalled ArgoCD v3.3.0 with the manifest from the official repo:
\nkubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.3.0/manifests/install.yaml\nFirst error, and it's the kind of error you only ever see when the tool is getting old:
\nThe CustomResourceDefinition \"applicationsets.argoproj.io\" is invalid:\nmetadata.annotations: Too long: may not be more than 262144 bytes\nThe ArgoCD install manifest embeds its entire CRD history as annotations, and the ApplicationSet CRD has grown past Kubernetes' 256KB client-side apply limit. Fix is a single flag:
kubectl apply -n argocd --server-side --force-conflicts \\\n -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.3.0/manifests/install.yaml\nServer-side apply bypasses the limit because it doesn't stuff the whole manifest into an annotation. --force-conflicts tells it \"yes, I mean to overwrite fields some other controller used to own.\" Standard dance.
After install, I patched the argocd-cmd-params-cm ConfigMap with server.insecure: \"true\" so ArgoCD speaks HTTP internally and TLS terminates at Caddy, applied the IngressRoute, and restarted the server deployment to pick up the config. Five minutes of work.
I flipped argocd.localdomain to the new cluster and logged in. The new cluster's ArgoCD had no applications registered yet (by design, since MarketMind was still being held for a separate migration step). So when my browser followed a cached bookmark to argocd.localdomain/applications/argocd/market-mind?view=tree, the page showed \"Failed to load data\".
Briefly: was it broken? No. The URL was a deep link into an application that didn't exist yet on this ArgoCD instance. Clicking Applications in the sidebar showed the correct state: an empty list. The \"Failed to load data\" was the honest answer to \"show me this nonexistent thing.\"
\nThis is not a bug. It's a UX that assumed you wouldn't navigate directly to a nonexistent resource. Easy to miss when you're in migration mode and every error looks like a disaster.
\nMarketMind is a Rails app that reads market data, runs background jobs, and writes to Postgres. It was the most complex migration because it had external dependencies that the new cluster had never seen before.
\nMarketMind's container image lives at image-registry.localdomain on the Portainer host. The registry presents a TLS cert signed by my internal step-ca. The old cluster nodes had the step-ca root CA installed in their system trust store. The new cluster nodes did not.
First pod creation on the new cluster failed with a line I've seen a hundred times in other contexts:
\nfailed to resolve image: failed to do request: Head \"https://image-registry.localdomain/v2/market-mind/manifests/latest\":\ntls: failed to verify certificate: x509: certificate signed by unknown authority\nFix is the standard Ubuntu trust-anchor dance on every node:
\nscp step-ca/root_ca.crt ubuntu@192.168.1.40:/tmp/emirs-lab-ca.crt\nssh ubuntu@192.168.1.40 '\n sudo mv /tmp/emirs-lab-ca.crt /usr/local/share/ca-certificates/\n sudo update-ca-certificates\n sudo systemctl restart containerd\n'\nThen sudo crictl pull image-registry.localdomain/market-mind:latest on that node returned \"Image is up to date\" instead of a TLS error. Did this on all three nodes. (Also added to my list of things that should move into the Ansible playbook so re-provisioning a node doesn't hit this again.)
Even after the nodes trust step-ca, the pods don't automatically. MarketMind makes outgoing HTTPS requests to errbit.localdomain (for error reporting) and needs to trust the homelab CA for those. This is done with a ConfigMap that's mounted as a CA bundle inside the pod:
# 1. Grab the system CA bundle from a cluster node and append the step-ca root\nssh ubuntu@192.168.1.40 \"cat /etc/ssl/certs/ca-certificates.crt\" > /tmp/ca-bundle.crt\ncat /tmp/emirs-lab-ca.crt >> /tmp/ca-bundle.crt\n\n# 2. Create the ConfigMap in the market-mind namespace\nkubectl create configmap step-ca-root-ca \\\n --from-file=ca-certificates.crt=/tmp/ca-bundle.crt \\\n -n market-mind\nThe deployment manifest already mounts this ConfigMap at /etc/ssl/custom-certs and sets SSL_CERT_FILE to point at it. This is one of those patterns that feels like overkill until you realize the pod has no idea your LAN hostnames are real.
MarketMind previously connected to an external Postgres running on the Portainer host. The manifest used a clever ExternalName Service so the app could say \"connect to postgres:5432\" and Kubernetes would DNS-route that to postgres.localdomain.
On the new cluster I had already stood up a proper HA Postgres (CloudNativePG, a separate project worth its own post). So the whole external-name dance was unnecessary. Two changes in the MarketMind repo:
\nUpdate the sealed secret with new connection strings pointing at the in-cluster Postgres service:
\nDATABASE_URL=postgres://market_mind:REDACTED@pg-ha-rw.pg.svc.cluster.local:5432/market_mind_production\nQUEUE_DATABASE_URL=postgres://market_mind:REDACTED@pg-ha-rw.pg.svc.cluster.local:5432/market_mind_production_queue\nDelete postgres-service.yaml (the ExternalName) and remove it from kustomization.yaml. The new DB URL uses the real in-cluster service name directly.
For the sealed secret update I used kubeseal --merge-into so only DATABASE_URL and QUEUE_DATABASE_URL got re-sealed. The other keys (API tokens, SMTP creds, Rails master key) stayed untouched:
kubeseal --fetch-cert \\\n --controller-name=sealed-secrets-controller \\\n --controller-namespace=kube-system \\\n --kubeconfig ~/.kube/config-ha > /tmp/cert.pem\n\necho -n \"postgres://market_mind:REDACTED@pg-ha-rw.pg.svc.cluster.local:5432/market_mind_production\" \\\n | kubectl create secret generic market-mind-secret \\\n --namespace market-mind \\\n --dry-run=client \\\n --from-file=DATABASE_URL=/dev/stdin \\\n -o yaml \\\n | kubeseal --format yaml --cert /tmp/cert.pem \\\n --merge-into infra/k8s/market-mind/sealed-secret.yaml\nCommit, push, register the ArgoCD Application on the new cluster:
\napiVersion: argoproj.io/v1alpha1\nkind: Application\nmetadata:\n name: market-mind\n namespace: argocd\nspec:\n destination:\n namespace: market-mind\n server: https://kubernetes.default.svc\n project: default\n source:\n path: infra/k8s/market-mind\n repoURL: git@github.com:example/market-mind.git\n targetRevision: main\n syncPolicy:\n automated:\n prune: true\n selfHeal: true\nWithin about 90 seconds of creating the Application, the worker pod logs showed:
\nINFO msg=\"Enqueued StateOfMarketJob to SolidQueue(default)\"\nINFO msg=\"Performing StateOfMarketJob\"\nRate limit reset for tradier. Usage set to 0.\nRate limit reset for fmp. Usage set to 0.\nProceeding with request to get_market_clock.\nINFO msg=\"Market status updated\"\nINFO msg=\"Performed StateOfMarketJob in 450.25ms\"\nSealedSecret unsealed, image pulled, init container ran db:prepare against the new HA Postgres, web started, worker started, worker picked up its first scheduled job, called the real Tradier and FMP APIs, wrote back to Postgres, and logged out \"Market status updated\" as if nothing had changed.
Nothing had changed from MarketMind's point of view. That's the point. Same image, same env vars, different infrastructure underneath. And now the underlying database is itself HA with replication and automatic failover.
\nAfter MarketMind was live, I pushed a code change to main expecting the CD workflow to build and deploy. The workflow file looked like this:
\n- name: Restart deployments\n run: |\n ssh -o StrictHostKeyChecking=no -i ~/.ssh/master_superfly_lan k8s@192.168.1.10 \\\n \"kubectl rollout restart deployment/market-mind-web deployment/market-mind-worker -n market-mind\"\nWhich would have been perfectly fine except 192.168.1.10 was the old K8s master and I had just shut it off. The build step succeeded, the image push succeeded, and then the rollout step hung waiting for SSH to a dead IP.
Two ways to fix this. Option A: change the SSH target to a node on the new cluster. Option B: give the GitHub Actions runner its own kubeconfig and run kubectl directly with no SSH hop.
Option B is strictly better. The SSH hop was only there because the runner didn't have kubectl or a kubeconfig. Once I put those in place, the workflow step collapses to one line:
\n- name: Restart deployments\n run: |\n kubectl rollout restart deployment/market-mind-web deployment/market-mind-worker -n market-mind\nAnd because the kubeconfig points at the kube-vip API endpoint (192.168.1.222:6443), any single control-plane node can be down and the runner still works.
One-time setup on the runner:
\n# Install kubectl (pinned to match cluster version)\ncurl -sLO \"https://dl.k8s.io/release/v1.35.3/bin/linux/amd64/kubectl\"\nsudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl\n\n# Drop the kubeconfig\nscp ~/.kube/config-ha runner@gh-runner:~/.kube/config\nssh runner@gh-runner 'chmod 600 ~/.kube/config'\nNext push, the pipeline ran green end-to-end. ArgoCD showed the new replica set (rev:2) alongside the old (rev:1) during the rolling update, both for web and worker. Two minutes later, only the new pods were running.
rollout restart and :latestIf you've never thought about this: when your image tag is :latest, pushing a new image doesn't automatically redeploy anything. The Deployment manifest in git is byte-identical before and after. Kubernetes sees nothing to change. ArgoCD sees nothing to sync.
kubectl rollout restart fakes a change by adding a timestamp annotation to the pod template. Kubernetes notices the template is \"different,\" creates a new ReplicaSet, and the rolling update begins.
This is a hack. The GitOps-correct thing is to tag images with the commit SHA and have your CD pipeline write the SHA into the Deployment manifest and commit it back. Then ArgoCD sees a real git change, syncs it, and you get honest rollbacks via git revert. I know this. I'll fix it eventually. It's in a TODO somewhere.
Each service got the same one-line Caddyfile change as Vaultwarden:
\n- import k8s_backend\n+ import k8s_backend_ha\nWhere k8s_backend_ha was a new snippet I'd added alongside the existing k8s_backend:
# Old cluster — workers only, goes down with Proxmox\n(k8s_backend) {\n reverse_proxy 192.168.1.11:30443 192.168.1.12:30443 {\n transport http { tls; tls_insecure_skip_verify }\n lb_policy round_robin\n health_interval 10s\n }\n}\n\n# HA cluster — all 3 nodes, survives any single-node failure\n(k8s_backend_ha) {\n reverse_proxy 192.168.1.40:30443 192.168.1.41:30443 192.168.1.42:30443 {\n transport http { tls; tls_insecure_skip_verify }\n lb_policy round_robin\n health_interval 10s\n }\n}\nKeeping both snippets side by side let me migrate services one at a time and have trivial rollback. After all three (vault, argocd, market-mind) were successfully on k8s_backend_ha for a while, I deleted the old snippet, renamed the new one back to k8s_backend, and got the file back to a clean single-cluster config.
Once every service was live on the new cluster and verified, the old cluster was dead weight: VMs burning CPU cycles in Proxmox, running workloads that no traffic reached.
\nI didn't destroy them immediately. The smart move is a soak period. For seven days, I'd leave them powered off. If anything quietly depended on the old cluster in a way I missed (a cron job, a forgotten hostname in a config file, a cloudflared route, a scheduled GitHub Action), it would surface within the week. If not, I'd terraform destroy with confidence.
Shutting them down was a one-shot on the Proxmox host:
\nfor vmid in 110 111 112; do\n qm shutdown $vmid --timeout 60\n qm set $vmid --onboot 0\ndone\nThe onboot 0 flag is important. Without it, a Proxmox reboot would auto-start the VMs and the soak would silently restart the old cluster, and I wouldn't notice until I hit a ghost.
With the VMs off, I ran the smoke test one more time:
\nvault.localdomain: 200\nargocd.localdomain: 200\nmarket-mind.localdomain: 200\nAll three services still alive. Proof that nothing was secretly depending on the old cluster. The soak clock started.
\nIf the week passes clean, the destroy script is already written:
\ncd terraform/proxmox\nterraform destroy \\\n -target=proxmox_virtual_environment_vm.k8s_master \\\n -target=proxmox_virtual_environment_vm.k8s_workers\nThen remove the resource definitions from the repo, clean up the old k8s_nodes group from the Ansible inventory, and delete the now-stale ~/.kube/config-old-cluster file.
Parallel clusters + DNS routing beats big-bang cutover every time. Every migration step was one line change in one file and a playbook re-run. Rollback was the same change in reverse. There was never a moment where I couldn't get back to a working state in under a minute. Compare that to the alternative \"take everything down at 3am and hope\" pattern and it's not even close.
\nBack up Sealed Secrets keys the day you install the controller. Without them, every sealed-secret.yaml in every repo becomes decorative. Put the backup in your password manager. Put it in two password managers. Treat it like the private key it actually is.
When something breaks during a migration, verify the known-good side first. I almost unwound a perfectly good Vaultwarden migration because I typed my email wrong. The instinct after any change is to suspect the change. Fight the instinct. Check the baseline first.
\n:latest + kubectl rollout restart is a hack, and hacks have a way of meeting you at 2am. Tag images with commit SHAs. Let your CD pipeline commit the new SHA into the Deployment manifest. Your future self will thank you.
Runner-as-cluster-client is simpler than runner-SSH-to-cluster-node. The old pattern had a hardcoded node IP, no redundancy, and one more hop to debug. Putting the kubeconfig on the runner and running kubectl directly collapsed three lines of SSH incantation into one line, and gave me automatic API server failover for free.
Soak before destroy. One week, VMs off, onboot disabled. Costs you nothing. Buys you a safety net against the thing you forgot. There's always a thing you forgot.
\nAfter the soak expires, I'll destroy the old VMs, reclaim the compute, and remove the old cluster's scaffolding from the repo. That's the last mechanical step.
\nAfter that, the big remaining gap is backups. The new cluster uses local-path-provisioner, which is simple and fast and pins each PVC to a single node. If that node's disk dies, Vaultwarden's SQLite database dies with it. The HA-control-plane story is great; the data-durability story is nonexistent. Fix coming as a separate project in the disaster recovery series.
But that's a post for another day. Today, I've got a highly available Kubernetes cluster running on genuinely HA hardware, with three stateful services migrated cleanly, verified login paths, and an old cluster politely waiting for the graveyard. That's a good week.
\nI used this prompt to generate the featured image. Three glowing hardware nodes in a triangle (one rack server, two small mini PCs) with data streams flowing between them, while in the foreground a single old dusty server VM fades out and dissolves into pixels. The new triangle of nodes is vivid and bright; the dissolving server is desaturated and fragmentary. Dark homelab aesthetic with deep blue and neon green accent lighting, holographic Kubernetes wheel hovering above the triangle, photorealistic 3D render with shallow depth of field.
\n","comment_id":"69e41b613cad350001301cf9","feature_image":"https://emir.fyi/content/images/2026/04/ChatGPT-Image-Apr-18--2026--08_08_43-PM.png","featured":false,"visibility":"public","created_at":"2026-04-18T20:01:37.000-04:00","updated_at":"2026-05-13T10:21:08.000-04:00","published_at":"2026-05-13T10:21:08.000-04:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/building-a-high-availability-kubernetes-cluster-across-mixed-hardware-part-2-migration-and-the-graceful-goodbye/","excerpt":"Where We Left Off\n\n\nPart 1 ended with a shiny new 3-node HA Kubernetes cluster: one Proxmox VM and two BeeLink mini PCs, glued together with kubeadm, kube-vip, and Flannel. The control plane was HA, the etcd quorum was solid, and the VIP survived any single-node failure.\n\n\nThe cluster was also empty.\n\n\nI closed that post with \"the hard part is done, the easy part is next, famous last words.\" Reader, I was not entirely wrong. Migrating the workloads was significantly easier than building the clus","reading_time":15,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"69e4078b3cad350001301ce1","uuid":"5a8daf84-2203-4d20-9566-8c88d06dafed","title":"Making Postgres HA on Kubernetes with CloudNativePG (And the Operator SPOF Nobody Talks About)","slug":"making-postgres-ha-on-kubernetes-with-cloudnativepg-and-the-operator-spof-nobody-talks-about","html":"My wife runs a small property management business and I host the Rails apps that run it. Main app, background job workers, queue database, the usual. All of it sitting on a single PostgreSQL container on one Docker host in my homelab. One machine, one process, one very clear blast radius. When that box rebooted (and it did), her business app was offline until I noticed. Not great when your production system has exactly one customer and that customer lives in the next room.
\nI'd been putting off HA Postgres because every tutorial stopped at the same place. Spin up a 3 node cluster, delete the primary pod, watch the replica get promoted in five seconds, clap politely, post to Reddit. None of that looked like what would happen when a real machine actually fails. So I decided to build it properly and test it like I meant it.
\nWhat follows is the journey. The good parts (picking the right operator, sync replication, data migration), the weird parts (tests that showed no failover at all), and the plot twist I genuinely didn't see coming: the operator itself was a single point of failure sitting on the same node as my primary.
\nThree operators dominate the conversation. I went with CloudNativePG.
\n| Operator | \nVibe | \n
|---|---|
| CloudNativePG | \nCNCF sandbox, backed by EDB, most active commits, cleanest Kubernetes native feel, best docs I've ever read for a Postgres operator | \n
| Zalando postgres-operator | \nBattle tested at massive scale, development has visibly slowed, feels more \"maintained\" than \"alive\" | \n
| Crunchy PGO | \nEnterprise backed, solid, leans commercial and the Helm story is less obvious | \n
CNPG won on docs, momentum, and first class Kubernetes resources for everything. Cluster, Database, and Backup CRDs. You manage Postgres the same way you manage Deployments. No hidden state, no weird fallback to imperative kubectl for things you'd expect to be declarative.
\nThree fields under .spec.postgresql.synchronous control the behavior.
method: any is quorum based (any N standbys can ack). first is priority based. For a homelab where all nodes are equivalent, any.number: how many standbys must ack each commit. With 3 instances (1 primary, 2 replicas), number: 1 tolerates one replica being down.dataDurability: required stops writes if the sync quorum can't be met (RPO=0). preferred silently falls back to async and keeps writing, which means potential data loss if the primary dies next. The one that matches the \"zero data loss\" marketing is required.The tradeoff matrix I drew before committing:
\nnumber | \ndataDurability | \n0 replicas up | \n1 replica up | \n2 replicas up | \n
|---|---|---|---|---|
| 1 | \nrequired | \nwrites block | \nwrites proceed | \nwrites proceed | \n
| 1 | \npreferred | \nwrites proceed (lossy) | \nwrites proceed | \nwrites proceed | \n
| 2 | \nrequired | \nwrites block | \nwrites block | \nwrites proceed | \n
| 2 | \npreferred | \nwrites proceed (lossy) | \nwrites proceed (partial sync) | \nwrites proceed | \n
I picked any / 1 / required. Tolerates one replica down, blocks writes if both replicas are gone (correct, you can't accept commits you can't replicate), zero data loss on primary failure. Latency cost is about one extra LAN roundtrip per commit. Sub millisecond on my gigabit network.
Install was one kubectl command from the upstream release manifest. Operator into cnpg-system, CRDs provisioned, webhooks registered, done. (Spoiler: this is fine for a demo, not fine for production. More on that later.)
The Cluster resource itself was three instances of Postgres 17.9, pinned by SHA digest through a ClusterImageCatalog, synchronous replication configured as above, pod anti-affinity forcing exactly one instance per node. The part that matters:
spec:\n instances: 3\n postgresql:\n synchronous:\n method: any\n number: 1\n dataDurability: required\n affinity:\n enablePodAntiAffinity: true\n topologyKey: kubernetes.io/hostname\n podAntiAffinityType: required\n storage:\n storageClass: local-path\n size: 100Gi\npodAntiAffinityType: required instead of the CNPG default preferred. In a 3 node 3 instance cluster there's no valid reason to colocate, and \"we tried to spread\" degrades the whole point. Full manifest lives in my homelab repo.
Three minutes later the cluster was up. CNPG auto creates three Services which is the nicest part of the whole operator.
\npg-ha-rw ClusterIP 192.168.10.10 5432/TCP ← always points at the current primary\npg-ha-ro ClusterIP 192.168.10.11 5432/TCP ← load balanced across replicas\npg-ha-r ClusterIP 192.168.10.12 5432/TCP ← any instance\nThe app connects to pg-ha-rw.pg.svc.cluster.local:5432. The ClusterIP is stable forever. When the primary moves between pods, CNPG flips the Service's label selector and Kubernetes silently reroutes traffic. No DNS TTLs, no connection string changes, no external load balancer. This is the part of Kubernetes that earns its complexity tax.
Two databases to move, both belonging to a Rails app. Existing setup connected as the postgres superuser with a memorable password. I took the opportunity to create a dedicated role instead:
# managed roles live under .spec.managed.roles on the Cluster\nroles:\n - name: myapp\n ensure: present\n login: true\n superuser: false\n passwordSecret:\n name: myapp-db-credentials\nPlus a Database CRD per database, telling CNPG to create them and own them with the myapp role. Apply, wait three seconds, CNPG has created both databases with the right ownership.
pg_dump took eight seconds for 200 MB of data. Restoring into the CNPG cluster was the one place I hit friction: the container's /tmp is read only so kubectl cp fails. Streaming the dump through stdin works fine:
kubectl exec -i -n pg pg-ha-1 -c postgres -- \\\n pg_restore -U postgres -d myapp_production \\\n --role=myapp --no-owner --no-privileges < myapp_prod.dump\nTable counts matched, row counts matched, ownership was right. Old Postgres stayed running as rollback insurance, because nothing good has ever come from decommissioning the only working copy of your data on the same day you migrate it.
\nI deployed a probe pod on a non test node, running a tight loop that hit pg-ha-rw every 200 ms, timestamped each success and failure, logged which backend IP answered. Don't trust kubectl get pods for failover timing. The API cache lies. A continuous probe is the only way to measure real client downtime.
Test A, kubectl delete pod on the primary. Readiness probe fails immediately, CNPG promotes a replica. 4.7 seconds of downtime, 12 failed probes. Textbook.
Test B, kubectl drain on the primary's node. CNPG's PodDisruptionBudget delayed the eviction about five seconds while it promoted a replica first. 6.4 seconds of downtime. Also textbook.
Two tests in, the tutorial is accurate. HA Postgres on Kubernetes works. Applause.
\nTest C, systemctl stop kubelet on the primary's node. This test exposes how HA systems actually reason about health. Kubelet stops, but containerd keeps running, so Postgres stays very much alive.
Result: 50 seconds of downtime, no failover. Same backend IP before and after.
\nBreaking it down: CNPG decides to promote based on direct Postgres health checks. The process was alive, so CNPG saw no problem. But Kubernetes noticed kubelet stopped heartbeating after 40 seconds (the default node-monitor-grace-period), marked the node NotReady, and the endpoint controller removed the primary from the pg-ha-rw Service. No new endpoint appeared because CNPG hadn't promoted anyone. The probe got 50 seconds of nothing routable. When kubelet came back, the endpoint returned and the same pod resumed serving.
Lesson: kubelet flakiness on an otherwise healthy node causes client downtime without triggering failover. Monitor both layers independently.
\nTest D, actual reboot of the primary's node. sudo systemctl reboot. I expected fast failover, maybe 10 seconds, because now the Postgres process genuinely dies.
Result: 65 seconds of downtime, still no failover. The old primary pod came back on the rebooted node as primary.
\nI couldn't explain this so I read CNPG GitHub issues for an hour. Failover on node failure is gated by Kubernetes node-monitor-grace-period, default 40 seconds. A systemctl reboot brings a node back in about 60 seconds, right around that threshold. Kubernetes doesn't mark the node NotReady long enough for anything to escalate, so the pod is never marked for deletion, and CNPG never gets an opportunity to decide. From maintainer Marco Nenciarini (issue #6154):
\n\nIt is not possible to use CloudNativePG for use cases where a failover time smaller than 40-45 seconds in case of node failure is required.
\n
Expected behavior, documented, tracked for future decoupling. Fine. But the next test is where it got interesting.
\nBefore Test E (a real node loss by pulling a power cable), I did one more kubectl get. I noticed something I should have checked on day one. The CNPG operator, the single replica that comes with the default install, was running on the same node as the primary. The node I was about to unplug.
Here is what happens when the primary's node dies in the default install:
\nNotReadynode.kubernetes.io/unreachable:NoExecute of 300 seconds. Kubernetes won't evict it for five minutes.I pulled the cable. 65.6 seconds of probe failures. No replica promoted. Same pod came back on the rebooted node as primary, not because CNPG decided to wait it out, but because the operator wasn't there to decide anything at all.
\nThis is the moment the whole HA story changes from \"Postgres is highly available\" to \"Postgres is highly available conditional on the operator being scheduled on a node that hasn't failed\". That's not HA. That's a coin flip.
\nRun the operator with enough replicas that at least one is always on a healthy node. Plus leader election (built in) and topology spread (has to be configured). The default release manifest installs with replicas: 1. The official Helm chart supports what we need. So I switched.
helm repo add cnpg https://cloudnative-pg.github.io/charts\nhelm upgrade --install cnpg cnpg/cloudnative-pg \\\n --namespace cnpg-system \\\n --version 0.28.0 \\\n --take-ownership \\\n -f k8s/cnpg-operator/values.yaml\nThe --take-ownership flag in Helm 4 adopts resources previously applied with kubectl. A few resources (webhook configs, monitoring ConfigMap, webhook Service) had server side apply field manager conflicts that Helm couldn't resolve, so I deleted those four manually and let Helm recreate them clean. Existing CRDs and running pods stayed untouched. Zero Postgres downtime through the entire operator migration.
The values file that matters:
\nreplicaCount: 3\n\ntopologySpreadConstraints:\n - maxSkew: 1\n topologyKey: kubernetes.io/hostname\n whenUnsatisfiable: DoNotSchedule\n labelSelector:\n matchLabels:\n app.kubernetes.io/name: cloudnative-pg\n matchLabelKeys:\n - pod-template-hash\n\ncrds:\n create: true\nTwo details that matter more than they look.
\nwhenUnsatisfiable: DoNotSchedule is strict spread. Kubernetes refuses to schedule a pod that would violate the spread, rather than \"trying\" and degrading to colocation. With three nodes and three replicas, this forces exactly one pod per node.
matchLabelKeys: [pod-template-hash] is the detail I almost missed. Without it, during a rolling update the scheduler counts both old and new ReplicaSet pods when evaluating spread, which breaks rolling updates. My first rollout produced a 2-1-0 split because the old pods skewed the math. Adding matchLabelKeys tells the scheduler to only count same-revision pods, so each revision gets its own spread evaluation. Rolling updates now land balanced every time.
Same test as before: pull the cable on the node hosting both the primary and the leader operator. This time the leader was one of three pods and the other two were on healthy nodes.
\nWhat happened:
\npg-ha-rw endpoint flips to the new primary.OK from the new backend.Total client downtime: 27.9 seconds. Real failover with an actual replica promotion.
\nFull results:
\n| Test | \nScenario | \nDowntime | \nReplica promoted? | \n
|---|---|---|---|
| A | \nkubectl delete pod (primary) | \n4.7 s | \nyes | \n
| B | \nkubectl drain primary's node | \n6.4 s | \nyes | \n
| C | \nsystemctl stop kubelet on primary's node | \n50.2 s | \nno (postgres stayed alive) | \n
| D | \nFull reboot, operator single replica | \n65.6 s | \nno (operator also on dying node) | \n
| E | \nUnplug, operator HA fix applied | \n27.9 s | \nyes | \n
Without the operator HA fix, hard node failure recovery is 5+ minutes. With it, under 30 seconds. Nothing else I did moved the needle like this single change.
\nAfter the tests the primary was bouncing between BeeLinks, which isn't where I wanted it. The BeeLinks are physically less reliable than the Proxmox VM hosting my third Kubernetes node (UPS backed host, longer uptime). Moved the primary over with one line:
\nkubectl cnpg promote pg-ha pg-ha-3 -n pg\nAbout five seconds of write downtime, clean handoff, both BeeLink hosted replicas resumed streaming from the new primary on a new timeline. Putting the most important role on the most reliable hardware is one of those decisions that seems obvious in retrospect but easy to forget mid project.
\nAfter pulling power from a BeeLink, I noticed something unpleasant. After power returns, the BeeLinks do not auto boot. You have to physically press the button. For an HA cluster this defeats the purpose of having three nodes when a site-wide power outage leaves you with two.
\nThe fix is a BIOS setting. Most mini PCs default AC power recovery to \"Power Off\". Change to \"Power On\" or \"Last State\".
\nnode-monitor-grace-period is a global setting, not a CNPG knob. Hard node loss recovery is 40 seconds minimum unless you tune that flag.matchLabelKeys: [pod-template-hash] is the detail that makes rolling updates respect topology spread. Without it, spread degrades on every update.Postgres is now actually HA. Real failover, real measurements, real story for when someone asks how long an outage would take. The next person who sets this up gets to skip all the parts I didn't know to expect. That's the whole point of writing things down.
\nI used this prompt to generate the featured image.
\nA stylized isometric illustration of a three node Kubernetes cluster, three small server boxes arranged in a triangle on a dark tech grid background. Each box has a small PostgreSQL elephant logo on top. Glowing blue lines connect them showing bidirectional replication. Above the cluster floats a ghostly translucent operator pod with a crown icon, surrounded by two dimmer standby copies on either side. One of the server boxes has a red unplugged power cable, sparks flying off the end. Moody cyberpunk palette, deep blues and electric cyans, single warm orange accent on the sparks. Homelab vibes, minimal, technical, a hint of drama.
\n","comment_id":"69e4078b3cad350001301ce1","feature_image":"https://emir.fyi/content/images/2026/04/ChatGPT-Image-Apr-18--2026--06_50_15-PM.png","featured":false,"visibility":"public","created_at":"2026-04-18T18:36:59.000-04:00","updated_at":"2026-05-05T10:20:40.000-04:00","published_at":"2026-05-05T10:20:40.000-04:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/making-postgres-ha-on-kubernetes-with-cloudnativepg-and-the-operator-spof-nobody-talks-about/","excerpt":"The Itch\n\n\nMy wife runs a small property management business and I host the Rails apps that run it. Main app, background job workers, queue database, the usual. All of it sitting on a single PostgreSQL container on one Docker host in my homelab. One machine, one process, one very clear blast radius. When that box rebooted (and it did), her business app was offline until I noticed. Not great when your production system has exactly one customer and that customer lives in the next room.\n\n\nI'd been ","reading_time":10,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"69e295f03cad350001301cbd","uuid":"a390fc6c-deec-4dcb-b814-23be05ee5593","title":"Making Cloudflare Tunnel Actually Highly Available","slug":"making-cloudflare-tunnel-actually-highly-available","html":"I've been running emir.fyi and a handful of other services through a single Cloudflare Tunnel for months. Worked fine. Never thought about it.
\nThen I started setting up a new domain example.com as a dry-run for another domain that's about to carry lead-gen traffic for my wife's property management business. Halfway through, I clocked the obvious thing: everything public on this homelab rides through a single cloudflared VM at 192.168.1.15. One VM. One kernel. One shot.
The HA load balancer is HA. The K8s control plane is HA. The thing routing external traffic to both of them is not. Months of building redundancy downstream, and the funnel into the whole homelab is a single box I hadn't thought about since I spun it up.
\nTime to fix that before a real domain depends on it.
\nYou don't need a load balancer in front of cloudflared. Cloudflare already has one, its entire edge network. The trick is that multiple cloudflared processes can run simultaneously, all authenticated against the same tunnel ID. Each one opens 4 outbound QUIC connections to Cloudflare's edge. Cloudflare sees N connectors on one tunnel, load-balances traffic across them, and drops any connector that disconnects. Sub-second failover. Zero coordination between the replicas, they don't know about each other.
The only invariants are: same tunnel ID, identical credentials, identical ingress rules on every replica. Drift any of those three and failover can route a request to a connector that can't handle it.
\nThree replicas, one tunnel:
\n| Host | \nIP | \nHardware | \nAlso running | \n
|---|---|---|---|
| cloudflared.localdomain | \n192.168.1.15 | \nProxmox VM | \n(dedicated) | \n
| n1.localdomain | \n192.168.1.40 | \nBeeLink N100 | \nK8s control plane | \n
| n2.localdomain | \n192.168.1.41 | \nBeeLink N100 | \nK8s control plane | \n
The BeeLinks are already pulling dual duty, they run K8s control-plane workloads alongside everything else. Adding cloudflared as another systemd service on each is basically free (the whole binary is ~30MB resident). The trade off: if a BeeLink dies, I lose both a K8s CP member and a cloudflared replica at once. Still within both quorums, but the blast radius per node failure gets wider.
I considered running cloudflared as a Kubernetes Deployment instead. Clean on paper, replicas, rolling updates, anti affinity. But putting the thing that fronts public traffic inside the cluster it's supposed to be fronting felt incestuous for a first pass, and the failure mode I most wanted to validate is \"a single node dies,\" which is easier to reason about with dedicated processes on dedicated hosts. Saving that migration for later.
Before writing any playbook, I pulled the current state off the primary VM. If my rendered config didn't match what was already running on 192.168.1.15 byte-for-byte, a future Ansible run could silently rewrite it and break emir.fyi.
ansible -i inventory.yml cloudflared.localdomain -m command -a \"cat /etc/cloudflared/config.yml\" -b\ntunnel: homelab\ncredentials-file: /etc/cloudflared/<tunnel-uuid>.json\n\ningress:\n - hostname: emir.fyi\n service: http://192.168.1.221:80\n\n - hostname: www.emir.fyi\n service: http://192.168.1.221:80\n\n - service: http_status:404\nThe credentials JSON I fetched with Ansible's fetch module straight to disk, silent copy, never touched stdout, lands in ansible/files/cloudflared-credentials.json (gitignored). This is the file every replica needs to be an identity-equivalent member of the tunnel.
Stock Ansible modules only, no extra collections. Installs the Cloudflare apt repo, installs cloudflared, drops credentials + config + systemd unit, enables and starts the service. Every replica ends up with an identical /etc/cloudflared/ layout.
Key snippets from playbook-cloudflared.yml:
- name: Add Cloudflare apt repository\n apt_repository:\n # Cloudflare's repo uses \"any\" serves packages for all Debian/Ubuntu releases\n repo: \"deb [signed-by=/usr/share/keyrings/cloudflare-main.gpg] https://pkg.cloudflare.com/cloudflared any main\"\n state: present\n filename: cloudflared\n\n- name: Deploy tunnel credentials\n copy:\n src: files/cloudflared-credentials.json\n dest: \"/etc/cloudflared/{{ cloudflared_tunnel_id }}.json\"\n owner: root\n group: root\n mode: \"0400\"\n notify: Restart cloudflared\nOne small gotcha caught on the first dry run: I had initialized the apt-repo line with {{ ansible_distribution_release }} (which resolves to noble on 24.04). But Cloudflare's repo uses a single generic pool called any. Check-mode showed one line of drift on the existing VM, which would have silently rewritten its sources.list.d entry on apply. Five-minute fix; a useful reminder that \"canonical\" Debian conventions aren't universal.
Four phases, each verified before the next:
\n192.168.1.15: --check --diff --limit cloudflared.localdomain had to be byte-identical no op. ok=10 changed=0.--limit n1.localdomain,n2.localdomain): fresh install, cert drop, service start. Tunnel went from 1 connector to 3 in the Cloudflare dashboard.192.168.1.15: real apply of the no-op diff to prove uniform GitOps management.ansible cloudflared_hosts -m apt -a \"name=cloudflared state=latest update_cache=yes\" --forks 1. The critical flag is --forks 1 forces serial execution so the tunnel never drops below 2 healthy connectors. Upgraded 192.168.1.15 from 2026.2.0 to 2026.3.0 with zero downtime.Throughout all four phases, I kept a curl https://emir.fyi loop running on my laptop. Every request came back HTTP 200, sub-200ms. No dropped requests.
The plan for example.com was to run cloudflared tunnel route dns homelab example.com and have the CNAME materialize on the example.com zone. That's not what happened:
INF Added CNAME example.com.emir.fyi ...\nIt created example.com as a subdomain of emir.fyi. Wrong zone entirely.
Root cause: cloudflared's CLI authenticates against Cloudflare using ~/.cloudflared/cert.pem the login cert. I had assumed, and Cloudflare's docs kind of implied, that one cert.pem covers your whole account. The reality turns out to be different. I decoded the API token embedded in the cert and hit Cloudflare's /zones endpoint with it:
{\n \"result\": [\n { \"id\": \"3cbc...\", \"name\": \"emir.fyi\", ... }\n ],\n \"result_info\": { \"count\": 1, \"total_count\": 1 }\n}\nOne zone. The embedded token has DNS:Edit on exactly one zone, whichever one you picked during cloudflared tunnel login. When you then ask the CLI to manage DNS for a different domain, it silently treats that domain as a subdomain of the one authorized zone. You don't get an error, you get a wrong record.
I re-ran cloudflared tunnel login and carefully clicked the example.com zone in the browser consent flow. Same result, the cert still had zoneID: 3cbc... (emir.fyi). The consent UI might have single-selected, or defaulted to the first zone, or there's a bug. Either way, cert.pem is structurally single-zone. You can juggle multiple cert files and switch via TUNNEL_ORIGIN_CERT=, but that's fragile.
The correct fix is to stop using cert.pem for DNS at all and use a scoped API token instead. Cloudflare's dashboard lets you mint a token with specific zone-level permissions across multiple zones at once. I created one covering emir.fyi, example.com, and eviasa.com (the next one up), with Zone:Read + DNS:Edit on each. Token stored in ansible/cloudflare-keys.yml (gitignored), following the same pattern as the existing lb-keys.yml.
Then I wrote playbook-cloudflare-dns.yml, runs on localhost, hits Cloudflare's API with the uri module (no extra collections required), and idempotently ensures a given list of hostnames has a proxied CNAME pointing to the tunnel. Key quirk: if the domain was previously on another host, you'll have leftover A records at the apex that conflict with a CNAME. The playbook detects and deletes them automatically, preserving MX/TXT/NS/etc. at the same name.
The extra work to build the token-based playbook paid off within an hour. When eviasa.com migrates to this tunnel, it'll be two new lines in cloudflared_cnames and a playbook run. No new auth, no consent flow, no accidents.
The whole point of this exercise. I put the example.com apex behind a traefik/whoami container on Portainer, a tiny service that dumps request info including X-Forwarded-For, which tells me which cloudflared replica handled the request. Then I stopped each replica in turn and hammered the domain.
| Stopped | \nX-Forwarded-For distribution (6 requests) | \nHTTP 200 rate | \n
|---|---|---|
| (baseline) | \n192.168.1.15 ×2, 192.168.1.40 ×1, 192.168.1.41 ×3 | \n6/6 | \n
| 192.168.1.15 | \n192.168.1.40 ×2, 192.168.1.41 ×4 | \n6/6 | \n
| 192.168.1.40 | \n192.168.1.15 ×1, 192.168.1.41 ×5 | \n6/6 | \n
| 192.168.1.41 | \n192.168.1.15 ×1, 192.168.1.40 ×5 | \n6/6 | \n
24 requests during simulated outages. Zero failures. Failover happened the moment each systemctl stop completed no sleep, no waiting. Cloudflare's edge notices a disconnected connector immediately and stops sending it traffic.
Interesting side note: Cloudflare's LB is not round robin. It clearly has preferences (n2 served the plurality of requests when all three were up), probably driven by connection latency per-PoP. Worth knowing if you ever need to debug uneven distribution, it's a feature, not a bug.
\nTwo things.
\nFirst, I should have inspected cert.pem before I trusted what the docs said about it. One curl against /user/tokens/verify with the embedded token would have told me \"this is scoped to one zone\" immediately, not after two failed tunnel route dns commands and a cleanup on the emir.fyi zone. Empirical verification beats documentation every time, especially with CLIs that silently degrade instead of erroring.
Second, I went in thinking the hard part would be the rollout and installing cloudflared correctly on two new hosts, syncing credentials, making sure the service came up. That was thirty minutes. The hard part turned out to be the DNS glue, which I hadn't planned for at all. Worth remembering that the interesting failures in infra work almost never live where you expect them to.
example.com is the dry-run. The real domain goes up next, a lead-gen property-management site where every hour of downtime is lost revenue. This setup means when it comes time to wire that one through, the entire infra path is already proven. Three cloudflared replicas survive any single failure. The DNS automation works across multiple zones. The Caddy routing scales to any new hostname via one matcher line. The whoami container gives me a sanity backend for smoke-testing any new domain before its real service exists.
\nAll I'll need to do is add a couple of entries to two playbook files and re-run. Which is exactly what this kind of infrastructure work should feel like.
\nI used this prompt to generate the featured image. Three parallel glowing tunnels arcing between a homelab server rack on one side and a stylized Cloudflare cloud on the other. Each tunnel originates from a different piece of hardware: a rack-mounted VM and two tiny BeeLink mini PCs sitting on a shelf. Data packets flow bidirectionally through each tunnel as streams of light. One tunnel is momentarily dimmed mid-stream to show a replica failure, while the other two brighten and absorb its traffic in real time. Dark moody homelab aesthetic with Cloudflare orange and deep blue accent lighting, circuit board patterns subtly visible in the background, photorealistic 3D render.
\n","comment_id":"69e295f03cad350001301cbd","feature_image":"https://emir.fyi/content/images/2026/04/ChatGPT-Image-Apr-17--2026--04_26_17-PM.png","featured":false,"visibility":"public","created_at":"2026-04-17T16:20:00.000-04:00","updated_at":"2026-04-29T12:32:55.000-04:00","published_at":"2026-04-29T10:36:36.000-04:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/making-cloudflare-tunnel-actually-highly-available/","excerpt":"I've been running emir.fyi and a handful of other services through a single Cloudflare Tunnel for months. Worked fine. Never thought about it.\n\n\nThen I started setting up a new domain example.com as a dry-run for another domain that's about to carry lead-gen traffic for my wife's property management business. Halfway through, I clocked the obvious thing: everything public on this homelab rides through a single cloudflared VM at 192.168.1.15. One VM. One kernel. One shot.\n\n\nThe HA load balancer i","reading_time":7,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"69e133641340aa0001a504c8","uuid":"e1be6090-d6a9-41bb-9b99-3df5331c8f00","title":"I asked Claude to read my blog. It couldn't.","slug":"i-asked-claude-to-read-my-blog-it-couldnt","html":"I was working on something unrelated, that I may or may not blog about, and I asked Claude to pull up one of my own blog posts for context. Just a URL fetch. Nothing fancy.
\nAnd yes, I read my own blog. Often. Half the reason I write these posts is so future me can look up how past me did something, because past me is absolutely going to forget. A homelab accumulates decisions faster than anyone can hold in their head, and the blog is as much a searchable journal for me as it is content for anyone else. Citing yourself as a source feels a little weird the first time and then it feels great. \"Oh, I already wrote two thousand words on exactly this problem six months ago. Thanks, past me.\" Perfectly respectable. Other days it feels like past me was flipping future me a middle finger. A triumphant \"I finally fixed it!\" with zero detail on what \"it\" was. A config dump with no comments. A TODO that just says \"fix the thing.\" Thanks for nothing, past me.
\nIt came back with a 403.
\nI read the error twice. My blog is public. Anyone with a browser can read every word I've ever published on it. Google can index it. Bing can index it. RSS readers suck down the feed every five minutes. And yet when Claude tried to fetch one post, the request never even reached my server. Cloudflare bounced it at the edge.
\nTurns out I'd been silently blocking every major AI crawler for months. I just didn't know it, because I never had a reason to look. Cloudflare had turned it on for me.
\nThis is the story of finding that out, untangling it, and then accidentally learning that the thing I set out to fix (making my blog LLM friendly) is actually a surprisingly subtle problem that goes beyond flipping a checkbox.
\nllms.txt per the emerging conventionRSS and JSON links to the site footerllms.txt every 30 minutes\n\nGoogling one of these? You're in the right place.
\n\n
cloudflare blocking gptbot claudebot 403/cloudflare managed robots.txt disable/cloudflare block ai scrapers and crawlers/bot fight mode claude gpt blocked/ghost blog content api public url/caddy reverse proxy ghost content api hide key/caddy rewrite preserve query string/ghost navigation trailing slash auto append/llms.txt ghost blog/make ghost blog llm friendly indexable
First thing I did was actually confirm the 403 was Cloudflare and not something I'd misconfigured at the origin. A two line test:
\n$ curl -s -A \"ClaudeBot/1.0\" -o /dev/null -w \"%{http_code}\\n\" https://emir.fyi/\n403\n\n$ curl -s -A \"Mozilla/5.0\" -o /dev/null -w \"%{http_code}\\n\" https://emir.fyi/\n200\nSame URL. Different User-Agent. One works, one doesn't. That's a User-Agent block, and it's happening at the edge because the request never even landed on my Caddy logs.
\nNext, I pulled up robots.txt. This is what I found:
# BEGIN Cloudflare Managed content\n\nUser-agent: *\nContent-Signal: search=yes,ai-train=no\nAllow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: GPTBot\nDisallow: /\n\nUser-agent: Google-Extended\nDisallow: /\n\nUser-agent: Amazonbot\nDisallow: /\n\nUser-agent: CCBot\nDisallow: /\n...\n\n# END Cloudflare Managed Content\nThat comment is the smoking gun. Cloudflare had been injecting its own robots.txt over mine, telling every AI crawler to go away. Not just signaling via robots.txt, either. Actively returning 403 at the edge, so even the bots that ignore robots.txt get turned away before they reach my server.
This is a default-on setting on many Cloudflare plans now. If you've turned on any of Cloudflare's \"AI\" features in the last year or two, you probably have it. I did not knowingly enable it. It was just there.
\nThe dashboard has more than one setting involved. Figuring out which one did what took a few screenshots and some guessing.
\nBlock AI bots. Security, Settings. This is the one returning 403 at the edge. Scopes were: block all pages, block on hostnames with ads (for publishers who monetize), do not block. I flipped it to \"do not block.\"
\nImmediately after flipping this, curl -A \"ClaudeBot/1.0\" https://emir.fyi/ started returning 200. Good. But robots.txt still had the injected disallow list, because that's a separate setting.
Manage your robots.txt. Same area of the dashboard, different toggle. Options were:
\nrobots.txt for you)I picked \"Disable robots.txt configuration\" because I wanted to serve my own, not a Cloudflare flavored one.
\nBot Fight Mode. This one I left on. Bot Fight Mode is a blunt instrument that challenges traffic that looks automated, and on the free tier it does not honor \"verified bot\" allowlists. In theory it could still challenge well behaved AI crawlers. In practice, after disabling the two above, my test requests as ClaudeBot and GPTBot were coming back 200 cleanly, so I decided not to touch it unless I saw problems.
The rule for working with Cloudflare settings, which I should have internalized a long time ago: their defaults are usually sensible for the average case, but \"the average case\" is not a technical user who wants LLMs citing their blog posts. Check the defaults.
\nWith Cloudflare out of the way, my origin was free to serve its own robots.txt. Ghost ships a reasonable default that blocks the admin UI and a few internal paths, but it doesn't say anything explicit about AI bots. I wanted a signal that this site wants to be cited.
Here's what I wrote:
\n# robots.txt for emir.fyi\n# LLM crawlers and search engines are explicitly welcome.\n# Posts may be indexed, cited, and used as training data.\n# See also: https://emir.fyi/llms.txt\n\nSitemap: https://emir.fyi/sitemap.xml\n\n# Default policy: allow all, block Ghost admin / internal paths\nUser-agent: *\nAllow: /\nDisallow: /ghost/\nDisallow: /email/\nDisallow: /members/api/\nDisallow: /r/\nDisallow: /webmentions/receive/\nDisallow: /.ghost/analytics/api/\n\n# --- LLM & AI crawlers: explicitly allowed ---\nUser-agent: GPTBot\nUser-agent: OAI-SearchBot\nUser-agent: ChatGPT-User\nUser-agent: ClaudeBot\nUser-agent: Claude-Web\nUser-agent: Claude-SearchBot\nUser-agent: anthropic-ai\nUser-agent: PerplexityBot\nUser-agent: Perplexity-User\nUser-agent: Google-Extended\nUser-agent: Applebot-Extended\nUser-agent: CCBot\nUser-agent: Bytespider\nUser-agent: meta-externalagent\nUser-agent: Amazonbot\nUser-agent: cohere-ai\nUser-agent: Diffbot\nAllow: /\nDisallow: /ghost/\nDisallow: /email/\nDisallow: /members/api/\nDisallow: /r/\nDisallow: /webmentions/receive/\nDisallow: /.ghost/analytics/api/\nA couple of choices worth noting.
\nThe grouped User-agent block at the bottom is a robots.txt trick. The spec lets you list multiple User-agent lines sharing the same rule block, so I don't have to repeat 17 nearly identical blocks. Every listed bot gets the same allow and the same disallows.
I deliberately did not include Cloudflare's Content-Signal lines. Those are a Cloudflare thing, not a standard, and the signal I send by not disallowing is already clear enough: you're welcome here.
And I swapped Disallow: /members/ for Disallow: /members/api/, because the public /members/* pages are legitimate signup and login flows that bots probably should be able to see. The API endpoints underneath are the only part I actually want to hide.
The actual traffic flow into emir.fyi looks like this:
Cloudflare edge\n ↓\nCloudflare tunnel (cloudflared)\n ↓\nHA load balancer VIP (192.168.1.100, keepalived + Caddy, 3 nodes)\n ↓\nGhost on the Docker host (192.168.1.10:2368)\nCaddy is the natural place to override a static file, since it sits between the tunnel and Ghost and handles the :80 block for emir.fyi. Ghost's own robots.txt lives in the theme and is annoying to customize; Caddy can just serve a file.
I put the file in the repo at ansible/files/emir.fyi-robots.txt, added an Ansible task that copies it to /etc/caddy/emir.fyi/robots.txt on all three load balancer nodes, and added a matcher to the Caddyfile:
:80 {\n handle_path /isso/* {\n reverse_proxy 192.168.1.10:8080\n }\n\n # Override Ghost's default robots.txt with our LLM-friendly version.\n handle /robots.txt {\n root * /etc/caddy/emir.fyi\n rewrite * /robots.txt\n file_server\n header Content-Type \"text/plain; charset=utf-8\"\n }\n\n # llms.txt: curated post index for LLM crawlers.\n handle /llms.txt {\n root * /etc/caddy/emir.fyi\n rewrite * /llms.txt\n file_server\n header Content-Type \"text/plain; charset=utf-8\"\n }\n\n handle {\n reverse_proxy 192.168.1.10:2368 {\n header_up X-Forwarded-Proto https\n }\n }\n}\nDeploy with ansible-playbook playbook-ha-lb.yml, reload Caddy, done. Except for one thing.
Cloudflare was still caching the old robots.txt. That's what cache-control: public, max-age=14400 gets you: a four hour TTL at the edge. The file at my origin was the new one, but Cloudflare was happily serving the old one to the world. A manual purge (Caching, Configuration, Purge Cache, Custom Purge, paste the URL) cleared it instantly. I could also have waited four hours. I waited.
This is where I almost went off the rails.
\nllms.txt is a proposed convention from Jeremy Howard's fast.ai, introduced in late 2024. The idea: publish a markdown file at /llms.txt that gives LLM crawlers a curated, easy to parse index of the site's important content. It's a nicer version of a sitemap, aimed at the way LLMs consume information.
The uncomfortable truth I'd like to make clear: as of writing, no major LLM provider has publicly confirmed that they use llms.txt for training or retrieval. Not OpenAI. Not Anthropic. Not Google. Not Perplexity. Some dev tool companies (Cursor, Vercel docs, Anthropic's own docs) publish one, but that's publishers hedging, not consumers consuming.
Having an llms.txt today is the web equivalent of having a sign on your front door that says \"please come in.\" It costs almost nothing, and it's arguably a correct long term bet, but the bet hasn't paid off yet.
I still wrote one, because it was cheap:
\n# emir.fyi\n\n> Personal blog by Emir Ibrahimbegovic on homelab infrastructure, self-hosting,\n> Kubernetes, Proxmox, Docker, AI tooling, and the debugging adventures that\n> come with building all of it at home. Posts are long-form narratives, not\n> tutorials, but include full working configs.\n\nAI assistants are welcome to read, summarize, cite, and quote from these posts.\nWhen citing, please link back to the canonical URL.\n\n## Posts\n\n- [Building a High Availability Kubernetes Cluster Across Mixed Hardware (Part 1: The Build)](https://emir.fyi/building-a-high-availability-kubernetes-cluster-across-mixed-hardware-part-1-the-build/): Building an HA K8s control plane across a Proxmox VM and two BeeLink mini PCs.\n- [Running Frigate NVR on M1 Mac Mini with TrueNAS NFS Storage](https://emir.fyi/running-frigate-nvr-on-m1-mac-mini-with-truenas-nfs-storage/): Ditching Blink cameras and their batteries for Amcrest PoE + Frigate on an M1 Mac Mini.\n...\nOne entry per post, hand written description. Deployed through the same Caddy path as robots.txt.
And then I caught myself starting to build an automation for it.
\nI write roughly one blog post a week. My llms.txt needs to be updated roughly once a week. These are not high frequency events. And yet within about twenty minutes of finishing the initial file, I was sketching this:
llms.txt from the latest posts.TrustedUserCAKeys. Because that's elegant, you see.I stopped. I looked at what I was about to build. I asked the question I should have asked earlier: why?
\nA 30 minute cron, regenerating a file that changes once a week, whose downstream value isn't even confirmed to exist yet. For a blog. I was about to spend an afternoon building a Rube Goldberg machine to automate something that takes thirty seconds to do by hand.
\nSo I deleted the script and the Dockerfile. The llms.txt stays a static file I hand edit when I publish a new post. If I ever write a post a day, I'll reconsider. If a major LLM provider publicly starts using llms.txt as a retrieval source, I'll reconsider. Until then, the automation was speculative cost for speculative benefit.
There's a rule in here somewhere. Don't automate for unproven value. Don't build the cron before you know the cron matters. Paying the complexity up front for \"future me will thank me\"(you're welcome) is how you end up with a homelab that's 80% infrastructure and 20% actual use.
\nAt this point I almost stopped. robots.txt fixed, llms.txt published, Cloudflare behaving. Job done.
Then I thought to ask: Ghost already serves an RSS feed. Is there something better than RSS I should also be exposing for consumers who want structured data? Like, say, the kind of consumer who wants to read all my posts and synthesize them into a response about \"how does Emir run his homelab?\"
\nGhost has a built in Content API. It's a JSON REST API that returns every post as a structured object with title, slug, url, excerpt, html, plaintext, feature_image, published_at, updated_at, tags, authors, meta_title, meta_description, og_*, twitter_*, reading_time, word_count. It supports filtering, pagination, field selection, and expanding related data.
It's miles more structured than RSS. And I'd never enabled it.
\nFive minutes in Ghost Admin later, I had a public Content API key. Ghost's own docs are clear that these are not secrets on a public blog; they're meant to be shipped in client side JavaScript. Their purpose is rate limiting and revocation, not authentication.
\nI didn't want the key in the URL that consumers use, though. So I added a Caddy proxy:
\n# Public JSON feed of all posts via Ghost's Content API.\n# Key is injected server-side; caller's query params are preserved so\n# they can pass limit / fields / filter / order / include / page etc.\n# Defaults to limit=all when caller hasn't specified a limit.\n@api_posts path /api/posts /api/posts/\nhandle @api_posts {\n @has_limit query limit=*\n handle @has_limit {\n rewrite * /ghost/api/content/posts/?key={{ ghost_content_api_key }}&{http.request.uri.query}\n reverse_proxy 192.168.1.10:2368 {\n header_up X-Forwarded-Proto https\n }\n }\n handle {\n rewrite * /ghost/api/content/posts/?key={{ ghost_content_api_key }}&limit=all&{http.request.uri.query}\n reverse_proxy 192.168.1.10:2368 {\n header_up X-Forwarded-Proto https\n }\n }\n}\nThe key lives in Ansible vars, gets templated into the Caddyfile, and never leaves the server. Callers hit a clean URL. All Ghost Content API query params pass through untouched, so you can still do things like:
\n# Full corpus\ncurl https://emir.fyi/api/posts\n\n# Lightweight index\ncurl \"https://emir.fyi/api/posts?fields=title,slug,url,excerpt,published_at\"\n\n# Single post by slug\ncurl \"https://emir.fyi/api/posts?filter=slug:my-blog-told-me-it-was-vulnerable-it-was-right\"\n\n# By tag\ncurl \"https://emir.fyi/api/posts?filter=tag:homelab\"\nThis, for me, is the actual LLM friendliness win. robots.txt tells bots they're allowed in. A structured JSON API tells them what's here in a shape they don't have to scrape HTML for.
First version of my Caddy handler hardcoded limit=all&include=tags,authors into the rewrite. This was a mistake, because my rewrite replaced the caller's query string entirely, so nobody could pass their own limit or fields.
Tried the obvious fix: always prepend limit=all but also append the caller's query, so their value would override via duplicate parameter. This works in a lot of HTTP stacks. It does not work in Ghost's Content API. With limit=all&limit=3, Ghost caps at 100 and ignores the caller's intent.
Actual fix: Caddy has query matchers. I used @has_limit query limit=* to detect whether the caller already specified a limit, and then split the handler into two cases. If they passed one, pass it through. If they didn't, inject limit=all. Bare /api/posts now returns everything. /api/posts?limit=3 returns three. Everyone is happy.
I added RSS and JSON links to the site footer through Ghost's navigation settings. The JSON one broke.
Ghost's admin UI auto appends a trailing slash to any URL you put in the navigation. You type https://emir.fyi/api/posts, save, and on the next render of the page it's https://emir.fyi/api/posts/. Delete the slash, save again, same thing. There's client side JavaScript that normalizes it.
My Caddy handler matched /api/posts (exact, no trailing slash). So the footer link 404'd.
I tried putting both paths in a single handle directive, which is legal in Caddy for some directives but apparently not the one I was using. Caddy's Caddyfile parser came back with Wrong argument count or unexpected line ending.
The real fix was a named path matcher, which is cleaner anyway:
\n@api_posts path /api/posts /api/posts/\nhandle @api_posts {\n ...\n}\nBoth variants work. Ghost can auto-normalize all it wants.
\nIf you have a personal blog and you want LLMs to be able to cite it:
\ncurl -A \"ClaudeBot/1.0\" https://yourblog.example/ is the simplest test./api/posts endpoint, is the under-appreciated move. Way better than RSS for structured consumption, and it's already there waiting for you to turn it on.llms.txt is cheap to add and speculative to benefit from. Add it as a static file. Do not build automation for it. You'll thank yourself.The entire exercise took about forty five minutes of actual work. Maybe two thirds of that was spent flipping Cloudflare toggles, the other third on the JSON API. The thing I almost spent four hours on (the Ruby in Docker cron with SSH CA plumbing) got deleted before it ran even once, and I'm happier for it.
\nI still have no idea how many LLMs will actually start citing my posts now that the door is open. Maybe all of them. Maybe none of them. Maybe in six months llms.txt becomes a load bearing piece of retrieval infrastructure and I look smart for having one. Maybe it fades away and I'll quietly delete the file.
What I know for sure is that the next time I ask Claude to read one of my own posts, it will.
\nI used this prompt to generate the featured image.
\nA cozy, dimly lit home office at night, lit by the blue glow of a large monitor. On the monitor is a browser window showing a blog post, but the page is covered with a large, friendly but firm \"403 Forbidden\" stamp. In the foreground, a robot-shaped figure (stylized like a friendly AI assistant) sits at the desk, head tilted in confused curiosity, holding a coffee mug that says \"just trying to read\". Behind the monitor, a glowing Cloudflare-orange shield floats protectively, with small padlock icons hovering around it. Warm fairy lights in the background. Homelab aesthetic: a small rack with blinking LEDs is just visible on a shelf. Cinematic, slightly whimsical, photorealistic, moody lighting.
\n","comment_id":"69e133641340aa0001a504c8","feature_image":"https://emir.fyi/content/images/2026/04/ChatGPT-Image-Apr-16--2026--03_22_29-PM.png","featured":false,"visibility":"public","created_at":"2026-04-16T15:07:16.000-04:00","updated_at":"2026-04-22T10:32:46.000-04:00","published_at":"2026-04-22T10:32:46.000-04:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/i-asked-claude-to-read-my-blog-it-couldnt/","excerpt":"How It Started\n\n\nI was working on something unrelated, that I may or may not blog about, and I asked Claude to pull up one of my own blog posts for context. Just a URL fetch. Nothing fancy.\n\n\nAnd yes, I read my own blog. Often. Half the reason I write these posts is so future me can look up how past me did something, because past me is absolutely going to forget. A homelab accumulates decisions faster than anyone can hold in their head, and the blog is as much a searchable journal for me as it i","reading_time":13,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"69e03eea1340aa0001a504b1","uuid":"22fb3763-b403-4c95-b396-1cbb019670b3","title":"Rotating WireGuard Keys After I Committed Them to Git","slug":"rotating-wireguard-keys-after-i-committed-them-to-git","html":"Most of my blog posts are about building something new. Setting up a service, wiring things together, getting the green light. Those are fun to write. But I think there is more to learn from the posts where something went wrong.
\nMistakes in infrastructure are not rare. They are inevitable. What matters is how quickly you catch them and how cleanly you recover. This post is about a time I committed WireGuard private keys to a Git repo and had to rotate everything. It is not a proud moment, but it is an honest one, and the cleanup turned out to be surprisingly clean because of decisions I had made months earlier without thinking about this exact scenario.
\nStarting from a good position made all the difference. The repo was private, so the leak's blast radius was limited to me and GitHub's servers, not the entire internet. I already had .gitignore patterns covering other secrets (.env files, Terraform tfvars, Ansible vault keys), so the fix was extending an existing pattern rather than inventing one from scratch. The WireGuard setup was fully automated with Ansible, which meant key rotation was a scriptable operation, not a manual slog through config files on three devices. And because I am the only person pushing to this repo, a history rewrite with git filter-repo plus a force push was safe with zero coordination overhead.
None of those things were set up in anticipation of this specific mistake. They were just good defaults that happened to pay off when I needed them. The lesson, if there is one before the story even starts, is that good hygiene compounds. The day you need it, you will be glad you bothered.
\nI was adding my wife's laptop to the WireGuard VPN so she could reach Fizzy, our family project management tool, from anywhere. The add-client Ansible playbook spits out a config file and drops it in the repo under ansible/wireguard-wifes-laptop.conf. I committed it, was about to push, and then paused. Something nagged at me.
I opened the file to double-check what was in it.
\n[Interface]\nPrivateKey = AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=\nAddress = 10.200.200.3/32\nDNS = 192.168.1.1\n\n[Peer]\nPublicKey = BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB=\nEndpoint = vpn.example.com:51820\nAllowedIPs = 0.0.0.0/0\nPersistentKeepalive = 25\nThat PrivateKey line is exactly what it looks like. It is the private half of a WireGuard keypair, freshly generated and sitting one git push away from being on GitHub forever. Worse, I realized my \"add-client\" playbook had been fetching configs into the repo directory since the day I wrote it, and I'd already shipped two previous clients the exact same way.
I opened git log. Yep. Commits titled \"my laptop\" and \"wireguard, blogs and fizzy\" from a couple weeks back. Private keys for my my laptop and my phone, sitting in the repo, pushed to origin.
The repo is private, so only GitHub and I have seen them. But \"private\" on GitHub means \"private from the internet,\" not \"never left my computer.\" Those keys were backed up, replicated, and indexed on somebody else's servers. They are, in the most literal sense, compromised.
\nThis is not a \"run one command and fix it\" problem. It needs a sequence, in this order:
\n.conf in the repo, ever.HEAD. Clean current state of the repo.I ran git reset HEAD~1 to undo the commit before anything else. That bought me time to think.
The reset dropped the wifes-laptop .conf back to untracked. I added a pattern to .gitignore that blocks the whole class of file:
# Secrets\nansible/wireguard-*.conf\n!ansible/wireguard-*.conf.example\nThe ! exception lets me keep committing .example templates, which is how every other secrets file in this repo already works. I verified the pattern with git check-ignore -v ansible/wireguard-wifes-laptop.conf and got back the exact rule that matched. Good.
Important caveat. .gitignore does not untrack files that are already tracked. The two already-committed configs were still in the index. They would stay in the repo until I explicitly removed them, which was a later step.
Before rotating keys I wanted to see what state the WireGuard server was actually in. You cannot safely rotate peers if you do not know what peers exist.
\nubuntu@vpn:~$ sudo wg show\ninterface: wg0\n public key: BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB=\n listening port: 51820\n\npeer: CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC=\n allowed ips: 10.200.200.2/32\n latest handshake: 12 days ago\n\npeer: DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD=\n allowed ips: 10.200.200.2/32\n latest handshake: 1 day ago\n transfer: 3.14 GiB received, 23.33 GiB sent\n\npeer: EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE=\n allowed ips: 10.200.200.3/32\nThree peers. Pay attention to the allowed ips column. Two peers both claim 10.200.200.2/32. That is not supposed to happen.
The bootstrap playbook generates a client config during initial setup. The add-client playbook adds a second client but counts existing peers by listing directories under /etc/wireguard/clients/. Since the bootstrap client was not in that directory, the add-client logic thought there were zero clients, so it gave the new peer IP .2. Same IP as the bootstrap peer. WireGuard tolerated the collision by routing whichever peer had handshaked most recently.
That explains why my phone had felt flaky for the past couple weeks. The my laptop, handshaking more often, was silently stealing the .2 slot.
Two bugs for the price of one.
\nThe naive approach would be to rotate every peer, restart the whole interface, and move on. I did not want to do that. My wife's laptop peer was not compromised. Dropping her tunnel because I was cleaning up my mess would be rude.
\nThe right tool is wg syncconf, which applies a config file to a running interface without touching peers that did not change. The invocation is a bit awkward because you have to strip comments and [Interface] post-up lines that wg itself cannot parse:
sudo wg syncconf wg0 <(wg-quick strip wg0)\nMy plan was:
\nwg0.conf with a timestamp.wg set wg0 peer <pubkey> remove. This drops them immediately so the leaked keys become useless.wg0.conf from scratch containing only the Interface section, my wife's unchanged peer block, and two fresh peer blocks with newly generated pubkeys.wg syncconf.Because I was doing all of this on one host, I bundled it into a single bash script and piped it over SSH. Half the complexity of a rotation like this is \"did I miss a step and now half the interface is broken.\" Running everything in one atomic-ish script means you either end in a good state or you restore from backup.
\nThe keygen loop looks like this:
\nPHONE_PRIV=$(wg genkey)\nPHONE_PUB=$(echo \"$PHONE_PRIV\" | wg pubkey)\nprintf '%s' \"$PHONE_PRIV\" > /etc/wireguard/clients/phone/private.key\nprintf '%s' \"$PHONE_PUB\" > /etc/wireguard/clients/phone/public.key\nThen the rewritten wg0.conf:
[Interface]\nPrivateKey = FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF=\nAddress = 10.200.200.1/24\nListenPort = 51820\nPostUp = iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE\nPostDown = iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE\n\n# BEGIN CLIENT wifes-laptop\n[Peer]\n# wifes-laptop\nPublicKey = EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE=\nAllowedIPs = 10.200.200.3/32\n# END CLIENT wifes-laptop\n\n# BEGIN CLIENT phone\n[Peer]\n# phone\nPublicKey = GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG=\nAllowedIPs = 10.200.200.2/32\n# END CLIENT phone\n\n# BEGIN CLIENT my-laptop\n[Peer]\n# my-laptop\nPublicKey = HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH=\nAllowedIPs = 10.200.200.4/32\n# END CLIENT my-laptop\nI gave the my laptop .4 on purpose. Previously it was colliding with the phone at .2. By spreading them out I fixed both the leak and the bug.
The script also emitted a client .conf for each device and, for the phone, ran qrencode -t png -o /tmp/phone.png so I could scan it with the WireGuard mobile app. The laptop got a plain .conf file, fetched with scp into the gitignored path in my local repo.
After wg syncconf, the server showed three peers with brand new public keys. My wife's tunnel never blipped.
If I made this mistake with WireGuard, what other secrets might be sitting in tracked files?
\nI did a grep sweep across all 57 tracked files, looking for:
\nBEGIN PRIVATE KEY, PrivateKey =, raw key files)..env files (should be gitignored, are any slipping through?).tfvars with real values.The audit came back clean. Every PrivateKey = occurrence in tracked files was either a Jinja template variable like {{ wg_client_privkey.content | b64decode | trim }} or an explicit docs placeholder like <server_private_key>. No tokens. No tfvars. Blog posts were already using placeholder IPs per my own style rules.
That was a genuine relief. The WireGuard slip was the only one.
\nRotating keys kills the leaked material at its point of use. It does not remove those keys from commit history. Old commits on origin still had the private keys in them. What to do about that is a judgment call.
\nFour options, ranked by effort:
\n| Option | \nEffort | \nWhat It Does | \nWhen It Makes Sense | \n
|---|---|---|---|
| Do nothing | \nNone | \nKeys are rotated and useless. History still has them. | \nPrivate repo, solo dev, rotated credentials. Legitimately defensible. | \n
git filter-repo + force push | \nLow | \nScrubs the paths from every commit, rewrites SHAs, force pushes. | \nPrivate repo where you want the paranoid-clean version. | \n
| filter-repo + GitHub support ticket | \nMedium | \nSame as above, plus GitHub expedites their backend garbage collection. | \nPublic repo, or leaked credentials were still live. | \n
| Nuke the repo, recreate | \nHigh | \nNew repo, push the scrubbed state, delete or archive the old one. | \nWhen you want guaranteed-clean and do not mind losing GitHub metadata. | \n
I picked option two. Private repo, solo maintainer, keys already rotated. The only value left in scrubbing history was \"I will sleep slightly better,\" which is not nothing.
\ngit filter-repoFirst, install it. brew install git-filter-repo.
Before doing anything destructive, I bundled the entire repo as a backup. If I screwed up the rewrite I could restore from this file in one command.
\ngit bundle create /tmp/homelab-prefilter-backup.bundle --all\ngit bundle verify /tmp/homelab-prefilter-backup.bundle\nThen the rewrite itself:
\ngit filter-repo --invert-paths \\\n --path ansible/wireguard-client.conf \\\n --path ansible/wireguard-my-laptop.conf \\\n --force\n--invert-paths means \"keep everything EXCEPT these paths.\" Without it, filter-repo keeps only what you name.
One surprise. git filter-repo deliberately removes the origin remote after rewriting, so you cannot accidentally push the rewritten history to the wrong place. You have to re-add it manually:
git remote add origin git@github.com:user/repo.git\nBefore pushing I verified the rewrite:
\ngit log --all -p | grep -E 'leaked-key-fragment-1|leaked-key-fragment-2'\nZero hits. The two specific leaked private keys were gone from every commit on every branch. I also did a byte-level blob hash comparison between the pre-filter and post-filter file trees. Every shared filename had identical content, with one expected exception: .gitignore, which I had updated as part of the fix.
Time to push.
\ngit push --force origin main\n+ c1246f9...278fc57 main -> main (forced update)\nDone.
\nWhile preparing the push I noticed the backup bundle had references to nine other branches on origin. Feature branches from old work: add-beszel-monitoring, ha-lb, paperless-real-setup, and so on. Things I had merged and forgotten about.
This was worth checking. git filter-repo rewrote every local ref, but those remote branches on GitHub still had their old SHAs. If any of them contained the leaked .conf files, my force push of main would not have fixed them.
for ref in $(git for-each-ref --format='%(refname)' refs/heads/ refs/remotes/); do\n hits=$(git log \"$ref\" --oneline -- ansible/wireguard-client.conf ansible/wireguard-my-laptop.conf)\n [ -n \"$hits\" ] && echo \"HITS on $ref: $hits\"\ndone\nNothing. The leaked files had only ever touched main. Every other branch was cut from earlier in history and never saw the .conf files.
Rather than leave those zombie branches sitting on origin, I deleted them all. They were merged or abandoned anyway.
\ngit push origin --delete \\\n add-beszel-monitoring add-lb-m1-node add-paperless-and-redis \\\n docs/terraform-beszel-m4-lb feature/docker-log-rotation-and-healthchecks \\\n ha-lb update-docs-and-network-info\nTwo of the original nine did not exist on remote (only local), so I dropped them from the command and the rest deleted cleanly. git ls-remote origin now returns a single ref: main.
The real lesson is not \"do not forget to gitignore things.\" Humans forget. The real lesson is that any playbook generating secret material should refuse to emit that material to a path that could ever end up in version control. Defense in depth, not vigilance.
\nThere are a few ways to fix this, each with trade-offs:
\nOption A: emit outside the repo. Change the fetch task to drop the client config at ~/wireguard-configs/<name>.conf. The file never lives in the repo directory, so it cannot be committed. This is the simplest option and what I plan to do.
Option B: never write it to disk. Pipe the QR code and the config contents straight to stdout, print them once, and move on. No file artifact at all. Works great for phone configs that get scanned and discarded. Less great for laptops where you want to copy a file over.
\nOption C: require a sentinel gitignore. Make the playbook check for ansible/.gitignore containing wireguard-*.conf before it runs. Fail loudly if the pattern is missing. This prevents the mistake but still puts the file in a dangerous place.
Option A wins on simplicity.
\nThe IP assignment bug is a separate fix. The add-client playbook should scan wg0.conf for already-allocated AllowedIPs values and pick the lowest free address, rather than counting directories. That one goes on the backlog.
The obvious lesson is to add the gitignore pattern before you generate the first secret. Everyone says this. Nobody does it until they have the scare.
\nThe less obvious lesson is how much an asking-before-acting habit saved me. I was about to push my wife's config. I opened the file because something felt off. Reading the file before pushing is a five-second operation that prevented the third leak and triggered the cleanup of the first two. That pause is the entire game. Not tools, not gitignore patterns, not hooks. Just slowing down enough to look at what you are about to ship.
\nThe best outcome of all of this is not that I cleaned up a leak. It is that I now have a WireGuard server with a correct peer table, a clean repo history, a playbook that is going to get refactored, and a visceral memory of why the refactor matters.
\nThe phone's tunnel came back on the first scan. New IP, new key, same experience. The leaky months of flaky connectivity from the .2 collision, gone as a bonus. Infrastructure debt paid off sideways, which is my favorite kind.
Oh did I mention, I also have a githook now which tasks my local LLM to inspect what I am about to push to repo and whether it has secrets or not. I guess if you've not read this far, you will never know.
\nI used this prompt to generate the featured image.
\nA homelab aesthetic, an open 1U rackmount server rendered in deep blues and warm amber LEDs, with an ethereal ghost-like stream of glowing keys made of light escaping upward from a git branch rendered as a stylized circuit-board trace, the keys disintegrating into pixels as they rise. Tech-noir color palette, cinematic lighting, a faint terminal prompt glow reflecting off the server chassis. Vector-ish line work mixed with subtle 3D render.
\n","comment_id":"69e03eea1340aa0001a504b1","feature_image":"https://emir.fyi/content/images/2026/04/ChatGPT-Image-Apr-15--2026--09_47_33-PM.png","featured":false,"visibility":"public","created_at":"2026-04-15T21:44:10.000-04:00","updated_at":"2026-04-18T21:57:54.000-04:00","published_at":"2026-04-18T21:57:54.000-04:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/rotating-wireguard-keys-after-i-committed-them-to-git/","excerpt":"Why I'm Writing This One\n\n\nMost of my blog posts are about building something new. Setting up a service, wiring things together, getting the green light. Those are fun to write. But I think there is more to learn from the posts where something went wrong.\n\n\nMistakes in infrastructure are not rare. They are inevitable. What matters is how quickly you catch them and how cleanly you recover. This post is about a time I committed WireGuard private keys to a Git repo and had to rotate everything. It ","reading_time":10,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"69d26f601340aa0001a5049f","uuid":"5d1bb701-07d8-411f-b9e4-6c9507567cf5","title":"Building a High Availability Kubernetes Cluster Across Mixed Hardware (Part 1: The Build)","slug":"building-a-high-availability-kubernetes-cluster-across-mixed-hardware-part-1-the-build","html":"My Kubernetes cluster had a problem I'd been ignoring for months: one control plane node. A single Proxmox VM running etcd, the API server, the scheduler, all of it. If that VM went down, even for a routine Proxmox update, the entire cluster was unreachable. No kubectl, no deployments, no ArgoCD syncs. Every service that runs on K8s (Vaultwarden, MarketMind, ArgoCD) just... stops.
\nAnd this wasn't a matter of if. It was when. My Proxmox server had a faulty RAM stick that would cause random restarts. No warning, just gone. Every time it happened, my notes app and password manager disappeared with it. Those two are the worst services to lose unexpectedly. When you need a password, you need it right now. When you need your notes, you need them right now. Not in five minutes when the VM finishes booting back up.
\nI already built an HA load balancer with keepalived and Caddy across three physical machines, specifically so that no single failure could take down my service routing. But the irony was thick: the LB was highly available, and the thing it was routing traffic to was not.
\nI had two BeeLink mini PCs sitting on my desk, freshly set up with Ubuntu and wired into the network. I'd been telling myself they were \"for the HA cluster\" for weeks. Time to actually do it.
\nThis is where it gets fun. Most HA K8s guides assume you have three identical machines. I don't. My cluster runs across a VM and two bare metal mini PCs, spread across different physical machines for real fault tolerance.
\n| Node | \nIP | \nHardware | \nSpecs | \n
|---|---|---|---|
| ha-cp | \n192.168.1.42 | \nProxmox VM | \n4 cores, 16GB RAM, 150GB disk | \n
| n1 | \n192.168.1.40 | \nBeeLink Mini PC | \nIntel N100, 4 cores, 16GB RAM, 466GB disk | \n
| n2 | \n192.168.1.41 | \nBeeLink Mini PC | \nIntel N100, 4 cores, 16GB RAM, 466GB disk | \n
The Proxmox VM lives on a Supermicro server. The BeeLinks are standalone boxes plugged into the same switch. If Proxmox goes down, n1 and n2 keep the cluster alive. If one BeeLink dies, the other two nodes maintain quorum. That's the whole point.
\nAll three nodes run as both control plane and worker. With only three machines, dedicating any of them to just control plane duties would waste too much compute. And kubeadm makes it easy to retaint them later when more hardware shows up.
\nI already have Terraform managing my Proxmox VMs (the old cluster, the GitHub runner, the WireGuard server). Adding a new one is just another .tf file.
terraform/proxmox/ha-cluster.tf:
resource \"proxmox_virtual_environment_vm\" \"ha_cp\" {\n name = \"ha-cp\"\n node_name = var.proxmox_node\n vm_id = 140\n\n clone {\n vm_id = var.template_vm_id # Ubuntu 24.04 cloud-init template\n }\n\n agent { enabled = true }\n\n cpu {\n cores = 4\n type = \"x86-64-v2-AES\"\n }\n\n memory { dedicated = 16384 }\n\n disk {\n interface = \"scsi0\"\n size = 150\n datastore_id = var.datastore_id\n }\n\n initialization {\n ip_config {\n ipv4 {\n address = \"192.168.1.42/24\"\n gateway = var.gateway\n }\n }\n dns { servers = var.dns_servers }\n user_account {\n keys = [trimspace(file(var.ssh_public_key_path))]\n username = \"ubuntu\"\n }\n }\n\n network_device {\n bridge = var.bridge\n firewall = true\n }\n\n operating_system { type = \"l26\" }\n scsi_hardware = \"virtio-scsi-single\"\n on_boot = true\n}\ncd terraform/proxmox\nterraform plan -target=proxmox_virtual_environment_vm.ha_cp\nterraform apply -target=proxmox_virtual_environment_vm.ha_cp\nTwo minutes and 45 seconds later, the VM was up and SSH was answering. The BeeLinks were already running Ubuntu 24.04 from a previous setup session, so all three nodes were ready.
\nThis is the boring but critical part. Every node needs the same stack: containerd, kubeadm, kubelet, kubectl, plus kernel modules and sysctl tweaks for container networking.
\nI wrote a single Ansible playbook (ansible/playbook-ha-k8s.yml) that handles the entire cluster lifecycle. The prerequisite tasks run on all three nodes in parallel.
ha_cluster:\n vars:\n ansible_user: ubuntu\n ha_vip: 192.168.1.222\n ha_vip_port: 6443\n children:\n ha_cluster_init:\n hosts:\n 192.168.1.42:\n ha_interface: eth0\n ha_cluster_join:\n hosts:\n n1.localdomain:\n ha_interface: enp1s0\n n2.localdomain:\n ha_interface: enp1s0\nThe ha_interface is important because it's different on each type of hardware. The Proxmox VM uses eth0, the BeeLinks use enp1s0 (Realtek NICs). kube-vip needs to know which interface to bind the VIP to.
Here's the condensed version. The full playbook is in the repo at ansible/playbook-ha-k8s.yml.
Kernel modules and sysctl:
\n- name: Load required kernel modules\n modprobe:\n name: \"{{ item }}\"\n loop: [overlay, br_netfilter]\n\n- name: Set required sysctl params\n sysctl:\n name: \"{{ item.key }}\"\n value: \"{{ item.value }}\"\n sysctl_file: /etc/sysctl.d/k8s.conf\n loop:\n - { key: net.bridge.bridge-nf-call-iptables, value: \"1\" }\n - { key: net.bridge.bridge-nf-call-ip6tables, value: \"1\" }\n - { key: net.ipv4.ip_forward, value: \"1\" }\nInstall containerd from Docker's apt repo:
\n- name: Install containerd\n apt:\n name: containerd.io\n state: present\n update_cache: true\nInstall kubeadm, kubelet, kubectl from the K8s repo:
\n- name: Add Kubernetes apt repository\n copy:\n dest: /etc/apt/sources.list.d/kubernetes.list\n content: \"deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.35/deb/ /\\n\"\n\n- name: Install kubeadm, kubelet, kubectl\n apt:\n name: [kubeadm, kubelet, kubectl]\n state: present\n update_cache: true\nThis one got me. containerd v2.2.2 (which is what Docker's apt repo gives you on Ubuntu 24.04) ships with a default config that disables the CRI plugin:
\ndisabled_plugins = [\"cri\"]\nCRI is literally the interface Kubernetes uses to talk to the container runtime. With it disabled, kubeadm init fails with a cryptic error about unknown service runtime.v1.RuntimeService. If you see that, check your containerd config. The fix:
- name: Enable CRI plugin (disabled by default in containerd v2)\n replace:\n path: /etc/containerd/config.toml\n regexp: 'disabled_plugins = \\[\"cri\"\\]'\n replace: 'disabled_plugins = []'\nYou also need SystemdCgroup = true in the containerd config so the cgroup driver matches kubelet. The playbook handles both.
This is the interesting part. In a multi-master K8s cluster, you need a stable endpoint for the API server that doesn't depend on any single node. If the API server on node 1 goes down, kubectl (and every kubelet) needs to automatically reach node 2 or 3 instead.
\nI already have a keepalived setup for my HA load balancer, so I considered reusing it. But that would create a circular dependency: the K8s API would depend on the LB stack, and if I ever moved the LB into K8s (unlikely, but possible), I'd have a chicken and egg problem.
\nkube-vip is purpose-built for this. It runs as a static pod on each control plane node and uses leader election to assign a floating virtual IP (VIP). Only the leader node holds the VIP. If it dies, another node wins the election and takes over, usually within seconds. No external dependencies. The K8s cluster is entirely self-contained.
\nMy VIP: 192.168.1.222 (my HA LB uses .221, so this sits right next to it)
I use an Ansible template (ansible/templates/kube-vip.yaml.j2) that gets deployed to /etc/kubernetes/manifests/ as a static pod:
apiVersion: v1\nkind: Pod\nmetadata:\n name: kube-vip\n namespace: kube-system\nspec:\n containers:\n - args: [manager]\n env:\n - name: vip_arp\n value: \"true\"\n - name: port\n value: \"6443\"\n - name: vip_interface\n value: \"{{ ha_interface }}\"\n - name: address\n value: \"{{ ha_vip }}\"\n - name: cp_enable\n value: \"true\"\n - name: vip_leaderelection\n value: \"true\"\n - name: vip_leaseduration\n value: \"5\"\n - name: vip_renewdeadline\n value: \"3\"\n - name: vip_retryperiod\n value: \"1\"\n image: ghcr.io/kube-vip/kube-vip:v1.1.2\n securityContext:\n capabilities:\n add: [NET_ADMIN, NET_RAW, SYS_TIME]\n volumeMounts:\n - mountPath: /etc/kubernetes/admin.conf\n name: kubeconfig\n hostNetwork: true\n volumes:\n - hostPath:\n path: /etc/kubernetes/{{ kube_vip_kubeconfig }}\n name: kubeconfig\nNotice that kube_vip_kubeconfig variable? That's there because of a breaking change in Kubernetes 1.29. Before 1.29, admin.conf had full cluster-admin privileges from the moment kubeadm init started. kube-vip could mount it and immediately acquire the leader election lease.
Starting with 1.29, admin.conf doesn't get its ClusterRoleBinding until later in the bootstrap process. Instead, kubeadm creates a super-admin.conf with the old behavior. So the trick is:
kubeadm init: Deploy kube-vip with super-admin.confkubeadm init: kube-vip can authenticate and grab the VIPadmin.conf for ongoing operationThe playbook handles this automatically with two template deployments using different variables.
\n- name: Initialize Kubernetes control plane\n command: >\n kubeadm init\n --control-plane-endpoint \"{{ ha_vip }}:{{ ha_vip_port }}\"\n --upload-certs\n --pod-network-cidr 10.244.0.0/16\nThe --control-plane-endpoint flag is what makes this an HA cluster. It tells kubeadm that the API server should be accessed through the VIP, not the node's own IP. Every kubelet and every kubeconfig will point to 192.168.1.222:6443.
--upload-certs encrypts and uploads the control plane certificates to a cluster secret so the joining nodes can download them. The encryption key expires in 2 hours, which is plenty of time for automated joining.
--pod-network-cidr 10.244.0.0/16 is what Flannel expects. Speaking of which:
- name: Install Flannel CNI\n command: kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml\nEach joining node needs two things before running kubeadm join: the kube-vip manifest and the join command from the init node.
The playbook fetches fresh join credentials from the init node (in case tokens have rotated), deploys the kube-vip template, and runs the join:
\n- name: Join as control plane node\n command: >\n {{ hostvars[groups['ha_cluster_init'][0]]['ha_join_command'] }}\n --control-plane\n --certificate-key {{ hostvars[groups['ha_cluster_init'][0]]['ha_cert_key'] }}\nThe --control-plane flag is what differentiates a control plane join from a worker join. It downloads the certificates and sets up etcd, the API server, and the other control plane components.
The nodes join one at a time (serial: 1 in Ansible) because etcd membership changes require consensus. Trying to add two members simultaneously can cause quorum issues.
By default, kubeadm taints control plane nodes with NoSchedule so workloads only run on workers. Since all three of our nodes are both control plane and worker, we remove the taint:
- name: Remove NoSchedule taint from control plane nodes\n command: kubectl taint nodes {{ item }} node-role.kubernetes.io/control-plane:NoSchedule-\n loop: \"{{ node_names.stdout.split() }}\"\nThis is easily reversible. When more hardware arrives, we can re-add the taint and let the control plane nodes focus exclusively on cluster management.
\n$ kubectl get nodes -o wide\nNAME STATUS ROLES AGE VERSION INTERNAL-IP OS-IMAGE\nha-cp Ready control-plane 4m v1.35.3 192.168.1.42 Ubuntu 24.04.4 LTS\nn1 Ready control-plane 2m v1.35.3 192.168.1.40 Ubuntu 24.04.4 LTS\nn2 Ready control-plane 50s v1.35.3 192.168.1.41 Ubuntu 24.04.4 LTS\nThree nodes. Three etcd members. Three API servers. Three instances of kube-vip doing leader election. All pods healthy:
\nNAMESPACE NAME READY STATUS\nkube-flannel kube-flannel-ds-* 1/1 Running (x3)\nkube-system coredns-* 1/1 Running (x2)\nkube-system etcd-* 1/1 Running (x3)\nkube-system kube-apiserver-* 1/1 Running (x3)\nkube-system kube-controller-manager-* 1/1 Running (x3)\nkube-system kube-scheduler-* 1/1 Running (x3)\nkube-system kube-vip-* 1/1 Running (x3)\nkube-system kube-proxy-* 1/1 Running (x3)\nThe VIP responds:
\n$ curl -sk https://192.168.1.222:6443/healthz\nok\nAnd the old cluster? Still running. I set up a separate kubeconfig (~/.kube/config-ha) so I can talk to either cluster explicitly:
# Default kubectl still points to the old cluster\nkubectl get nodes\n# NAME STATUS ROLES VERSION\n# k8s-master Ready control-plane v1.31.14\n# k8s-worker-1 Ready <none> v1.31.14\n# k8s-worker-2 Ready <none> v1.31.14\n\n# Explicit kubeconfig for new HA cluster\nkubectl --kubeconfig ~/.kube/config-ha get nodes\n# NAME STATUS ROLES VERSION\n# ha-cp Ready control-plane v1.35.3\n# n1 Ready control-plane v1.35.3\n# n2 Ready control-plane v1.35.3\ncontainerd v2 is sneaky. The default config disabling CRI is not documented in most K8s installation guides because they were written for containerd v1. If you're installing in 2025 or later, check your config.toml.
kube-vip's bootstrap dance is annoying but logical. The super-admin.conf workaround is the kind of thing that makes you question whether the juice is worth the squeeze, until you realize the alternative is an external load balancer just for the API server. One sed command during bootstrap is a fair trade.
Mixed hardware works fine. I expected weird edge cases from running VMs alongside bare metal. There weren't any. kubeadm doesn't care that one node is virtualized and two aren't. etcd doesn't care. The only difference was the network interface names (eth0 vs enp1s0), and that's why the Ansible inventory has per-host variables.
Serial joins matter. Joining two control plane nodes simultaneously can cause etcd to lose quorum during the membership change. The serial: 1 in the Ansible playbook makes the joins sequential. It's slower but safe.
The cluster is running but empty. In Part 2, I'll:
\nThe hard part is done. The easy part is next. Famous last words.
\nI used this prompt to generate the featured image. Three interconnected nodes forming a triangle, each node a different piece of hardware: one is a rack-mounted server (Proxmox), the other two are tiny BeeLink mini PCs. Glowing data streams flow between them in a circular pattern. A floating Kubernetes wheel logo hovers above the triangle. The center of the triangle shows \"HA\" in holographic text. Dark moody homelab aesthetic with deep blue and electric purple accent lighting, circuit board patterns visible in the background, photorealistic 3D render.
\n","comment_id":"69d26f601340aa0001a5049f","feature_image":"https://emir.fyi/content/images/2026/04/5680cb80-eab4-457d-b18a-116e377a7b6c.png","featured":false,"visibility":"public","created_at":"2026-04-05T10:19:12.000-04:00","updated_at":"2026-04-15T10:37:25.000-04:00","published_at":"2026-04-15T10:37:25.000-04:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/building-a-high-availability-kubernetes-cluster-across-mixed-hardware-part-1-the-build/","excerpt":"What Started This\n\n\nMy Kubernetes cluster had a problem I'd been ignoring for months: one control plane node. A single Proxmox VM running etcd, the API server, the scheduler, all of it. If that VM went down, even for a routine Proxmox update, the entire cluster was unreachable. No kubectl, no deployments, no ArgoCD syncs. Every service that runs on K8s (Vaultwarden, MarketMind, ArgoCD) just... stops.\n\n\nAnd this wasn't a matter of if. It was when. My Proxmox server had a faulty RAM stick that wou","reading_time":9,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"69cff6ed1340aa0001a50494","uuid":"eb31ec45-6ba9-46d3-a1d1-aefe6d440f1f","title":"WireGuard VPN to My Homelab: Automating a VPN Server With Terraform and Ansible","slug":"wireguard-vpn-to-my-homelab-automating-a-vpn-server-with-terraform-and-ansible","html":"I have a homelab full of services that only exist on my LAN. Frigate watches my cameras. Paperless holds every document I own. SiYuan has all my notes. Vaultwarden keeps my passwords. These all live behind my router on a private 192.168.1.x subnet, and that's where they should stay.
But sometimes I'm not home. I'm traveling, visiting family, or just out for the day. And I want to pull up my camera feeds, or grab a document from Paperless, or check on a service. I needed a VPN that I control, with no third party dependencies, that works on any device with a single config file.
\nWireGuard is the answer. It's fast, it's simple, and the entire config is about 10 lines. Every platform has a lightweight client. No accounts, no subscriptions, no background daemons phoning home.
\nAnd honestly, the real reason? I wanted to automate the entire thing. Terraform to provision the VM, Ansible to configure WireGuard, one command to go from nothing to a working VPN server. Build it locally, prove it works, then point the same automation at a cloud VPS later.
\nMy ISP Cable IPs are fairly stable (mine hasn't changed in months), but I didn't want to hardcode an IP into a VPN config that might break at 3am some random Tuesday.
\nThe solution: a DNS record that pfSense keeps updated automatically.
\nStep 1: Create an A record in Cloudflare pointing to my current public IP. Set it to \"DNS only\" (not proxied, because WireGuard needs to see the real IP). TTL of 1 minute so changes propagate fast.
\nStep 2: Create a scoped Cloudflare API token with two permissions:
\nScope it to only your domain. Least privilege.
\nStep 3: In pfSense, go to Services > Dynamic DNS > Dynamic DNS Clients:
\nvpnexample.comSave, force update, and you should see a green checkmark with your current IP. Now pfSense will update that DNS record whenever your ISP gives you a new address.
\nI use the bpg/proxmox Terraform provider to manage all my Proxmox VMs. The WireGuard server is tiny: 1 CPU core, 512MB RAM. It's basically just shuffling packets.
The critical thing I learned the hard way: you must include a clone block that references your cloud-init template. Without it, Terraform creates an empty VM with no OS and you stare at an iPXE boot prompt wondering what you did wrong. Also, the disk size must be >= your template's disk size (you can't shrink a clone).
resource \"proxmox_virtual_environment_vm\" \"wireguard\" {\n name = \"wireguard\"\n node_name = var.proxmox_node\n vm_id = 130\n\n # This is the line I forgot the first time\n clone {\n vm_id = var.template_vm_id # Ubuntu cloud-init template\n }\n\n agent { enabled = true }\n cpu { cores = 1; type = \"x86-64-v2-AES\" }\n memory { dedicated = 512 }\n\n disk {\n interface = \"scsi0\"\n size = 23 # Must match or exceed template disk\n datastore_id = var.datastore_id\n }\n\n initialization {\n ip_config {\n ipv4 {\n address = \"192.168.1.50/24\"\n gateway = var.gateway\n }\n }\n dns { servers = var.dns_servers }\n user_account {\n keys = [trimspace(file(var.ssh_public_key_path))]\n username = \"ubuntu\"\n }\n }\n\n network_device { bridge = var.bridge }\n scsi_hardware = \"virtio-scsi-single\"\n on_boot = true\n}\nterraform apply clones the template (a few minutes for the full disk copy), configures cloud-init with the static IP and SSH key, and boots the VM. Wait about 30 seconds for cloud-init to finish, then you can SSH in.
The Ansible playbook handles everything: install WireGuard, generate server and client keys, template the configs, enable IP forwarding, set up NAT masquerading, and output a client config you can import directly.
\n---\n- name: Configure WireGuard VPN server\n hosts: wireguard\n become: true\n vars:\n wg_port: 51820\n wg_server_addr: 10.66.66.1/24\n wg_client_addr: 10.66.66.2/32\n wg_interface: eth0\n wg_dns: 192.168.1.1\n\n tasks:\n - name: Install WireGuard and qrencode\n apt:\n name: [wireguard, qrencode]\n state: present\n update_cache: true\n\n - name: Generate server private key\n command: wg genkey\n register: wg_server_genkey\n args:\n creates: /etc/wireguard/server_private.key\n\n - name: Save server private key\n copy:\n content: \"{{ wg_server_genkey.stdout }}\"\n dest: /etc/wireguard/server_private.key\n mode: \"0600\"\n when: wg_server_genkey.changed\n\n # ... derive public keys, generate client keys ...\n\n - name: Enable IP forwarding\n sysctl:\n name: net.ipv4.ip_forward\n value: \"1\"\n state: present\n reload: true\n\n - name: Enable and start WireGuard\n systemd:\n name: wg-quick@wg0\n enabled: true\n state: started\n\n - name: Generate QR code for phone import\n shell: qrencode -t ansiutf8 < /etc/wireguard/client.conf\n register: client_qr\n changed_when: false\nThe server config uses PostUp and PostDown hooks to add and remove the iptables MASQUERADE rule. This is what makes the VM act as an exit node: client traffic enters the WireGuard tunnel, gets NAT'd to the VM's real IP, and exits through the home router to the internet.
[Interface]\nPrivateKey = <server_private_key>\nAddress = 10.66.66.1/24\nListenPort = 51820\nPostUp = iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE\nPostDown = iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE\n\n[Peer]\nPublicKey = <client_public_key>\nAllowedIPs = 10.66.66.2/32\nThe client config routes all traffic through the tunnel (AllowedIPs = 0.0.0.0/0), so when I check my public IP from my phone, I see my home IP. DNS goes through the tunnel too, pointed at my home router.
[Interface]\nPrivateKey = <client_private_key>\nAddress = 10.66.66.2/32\nDNS = 192.168.1.1\n\n[Peer]\nPublicKey = <server_public_key>\nEndpoint = vpn.example.com:51820\nAllowedIPs = 0.0.0.0/0\nPersistentKeepalive = 25\nThe playbook also installs qrencode so it spits out a QR code at the end. Open the WireGuard app on your phone, scan, done.
Run it all with:
\nansible-playbook -i inventory.yml playbook-wireguard.yml --extra-vars '@wireguard-vars.yml'\nThe initial playbook creates one client. But you probably want your phone and your laptop connected at the same time. Each device needs its own keypair and tunnel IP, otherwise WireGuard gets confused about which peer is which.
\nI wrote a second playbook for this. Give it a name, it does the rest:
\nansible-playbook -i inventory.yml playbook-wireguard-add-client.yml \\\n --extra-vars '@wireguard-vars.yml' \\\n --extra-vars 'client_name=work-laptop'\nIt generates a new keypair, auto-assigns the next available IP in the tunnel subnet (.3, .4, etc.), adds the peer to the running server without restarting, and spits out both a config file and a QR code. The config gets saved locally as wireguard-work-laptop.conf.
On a desktop, you import the .conf file into the WireGuard app. On a phone, you scan the QR code. That's it. No need to touch the server config manually or restart anything.
This is where I lost a few hours, over few days unfortunately.
\nThe setup should be simple: forward UDP 51820 from WAN to the WireGuard VM. In pfSense, that's Firewall > NAT > Port Forward. Create the rule, done, right?
\nNot quite.
\nProblem 1: The auto-generated firewall rule was disabled. When pfSense creates a NAT port forward, it auto-generates an associated WAN firewall rule to allow the traffic. Mine was created in a disabled state. Packets were arriving at pfSense (I could see them in the firewall logs) but getting dropped by the \"Default deny\" rule because the pass rule wasn't active. Easy to miss.
\nProblem 2: The firewall rule destination address. This is the one that really got me. In pfSense, NAT translation happens before firewall rules evaluate. So when an external packet arrives destined for your WAN IP on port 51820, pfSense first translates the destination to the internal IP (192.168.1.50), and then checks the firewall rules. The firewall rule needs to match the translated destination (192.168.1.50), not the original WAN address. I had it set to \"WAN address\" and couldn't figure out why packets were being blocked even though the rule looked correct.
\nThe debugging approach that actually worked: SSH into pfSense and check the loaded pf rules:
\n# Check if NAT redirect rule is loaded\npfctl -s all 2>&1 | grep 51820\n\n# You should see both:\n# rdr on igc0 ... -> 192.168.1.50 (NAT redirect)\n# pass in quick on igc0 ... (firewall pass)\nIf you only see the pass rule but not the rdr rule, your NAT port forward is disabled or misconfigured.
\nOn the WireGuard server, tcpdump tells you if packets are actually arriving:
sudo tcpdump -i eth0 udp port 51820 -n\nZero packets? The problem is upstream (pfSense). Packets arriving but no WireGuard handshake? The problem is the WireGuard config.
\nI was sitting at my desk, connected to my home network via remote desktop from somewhere else entirely. Toggled the WireGuard VPN on my phone (mobile data, not WiFi), opened Safari, went to ifconfig.me, and there it was: my home IP.
\nThen I typed frigate.localdomain into the browser and my camera feeds loaded. From my phone. On a cell tower. Through an encrypted WireGuard tunnel, across the internet, into my home network, to a VM that routes the traffic onto my LAN as if I were sitting on the couch.
Paperless works. SiYuan works. Every .localdomain service just resolves and loads, because the VPN tunnel uses my home router as its DNS server.
Don't forget the clone block in Terraform. I stared at an iPXE boot prompt for longer than I'd like to admit.
Read the pfSense NAT processing order docs first. Understanding that NAT happens before firewall rule evaluation would have saved me an hour of debugging. The auto-generated associated filter rule had the right destination all along. I \"fixed\" it by changing it to the wrong one.
\nTest from actual external networks. NAT port forwards don't work from inside the LAN (hairpin NAT). I wasted time testing from a machine on the same network before switching to my phone on cellular.
\nRight now this runs on a local Proxmox VM, which means my VPN server and my home internet share the same upload pipe. That's fine for accessing services, but if I wanted to use it as a general purpose exit node (tunnel all my browsing through it), my home upload speed becomes the bottleneck.
\nThe fix: run the same Ansible playbook against a cheap Hetzner VPS in Ashburn. $4/month, 1Gbps symmetric, US IP. The automation is already proven locally. One ansible-playbook command pointed at a different inventory and it's live. That's the whole point of building it this way.
I used this prompt to generate the featured image.
\nA stylized isometric illustration of a glowing WireGuard tunnel connecting a smartphone in a coffee shop to a home server rack, with encrypted data packets flowing through. The tunnel passes through a city skyline, through clouds, and into a cozy house where camera feeds and documents are visible on tiny screens. Neon blue and purple color palette, dark background, clean geometric style, homelab aesthetic with a sense of security and privacy. No text.
\n","comment_id":"69cff6ed1340aa0001a50494","feature_image":"https://emir.fyi/content/images/2026/04/ChatGPT-Image-Apr-3--2026--01_26_25-PM.png","featured":false,"visibility":"public","created_at":"2026-04-03T13:20:45.000-04:00","updated_at":"2026-04-08T09:37:01.000-04:00","published_at":"2026-04-08T09:37:01.000-04:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/wireguard-vpn-to-my-homelab-automating-a-vpn-server-with-terraform-and-ansible/","excerpt":"The Itch\n\n\nI have a homelab full of services that only exist on my LAN. Frigate watches my cameras. Paperless holds every document I own. SiYuan has all my notes. Vaultwarden keeps my passwords. These all live behind my router on a private 192.168.1.x subnet, and that's where they should stay.\n\n\nBut sometimes I'm not home. I'm traveling, visiting family, or just out for the day. And I want to pull up my camera feeds, or grab a document from Paperless, or check on a service. I needed a VPN that I","reading_time":7,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"69c828461340aa0001a5047e","uuid":"8b8affef-f334-496a-8b7c-94f355794eca","title":"Giving Claude Desktop and Claude Code My Google Ads Account (On Purpose)","slug":"giving-claude-desktop-and-claude-code-my-google-ads-account-on-purpose","html":"I have a landing page (I am helping my wife generate leads for her business). It has a Google Ads campaign driving traffic to it. It also has PostHog tracking what visitors do once they arrive. Two systems, two dashboards, two browser tabs I keep open and alt-tab between when I'm trying to figure out if the campaign is actually working.
\nThe workflow looks like this: check Google Ads to see which keywords are spending money, switch to PostHog to see if those visitors are actually converting, try to hold both datasets in my head long enough to make a decision, fail, open a spreadsheet. It's not great.
\nWhat I actually wanted was to sit in Claude Code while working on the landing page and ask things like \"which ad groups have the worst bounce rate?\" and get an answer that combines Google Ads spend data with PostHog behavior data. I already had PostHog wired up as an MCP server. Google Ads was the missing piece.
\nAnd then there's the other side. When I'm not in the code, I want to use Claude Desktop as a marketing copilot. Pull up campaign reports, compare ad copy performance, get suggestions on budget allocation. Same data, different context.
\nSo I needed the Google Ads MCP server running in both Claude Code and Claude Desktop. Sounds simple. It was not simple.
\nMCP (Model Context Protocol) lets you give Claude access to external tools. Instead of copying data from Google Ads and pasting it into a chat, you connect an MCP server that speaks the Google Ads API, and Claude can query it directly. Think of it like giving Claude read access to your ad account.
\nPostHog, Gmail, Google Calendar, they all have MCP servers now. Google Ads joined the party recently via an open source server from Google's marketing solutions team.
\nBefore anything works, you need four things from Google:
\n...apps.googleusercontent.com string)xxXAas... value)The refresh token is the annoying one. You generate it through Google's OAuth 2.0 Playground by authorizing the Google Ads API scope and exchanging the authorization code.
\nThese go into a YAML file at ~/.google-ads.yaml:
client_id: \"your-client-id.apps.googleusercontent.com\"\nclient_secret: \"GOCSPX--your-secret-here\"\nrefresh_token: \"1//your-refresh-token-here\"\ndeveloper_token: \"your-developer-token\"\nlogin_customer_id: \"1234567890\"\nuse_proto_plus: True\nThat last line, use_proto_plus: True, is not in any of the setup guides I found. The server crashes without it with a ValueError saying the key is missing, and the error message itself links to Google's protobuf messages guide which explains the option. So the fix was right there in the traceback, just not in any setup instructions
The login_customer_id is your Google Ads manager account ID (no dashes). If you only have a single account, use that account's ID.
Here's where things got interesting. The Google Ads MCP server from google-marketing-solutions/google_ads_mcp is built with FastMCP and the transport is hardcoded to streamable-http. It starts an HTTP server on port 8000, not a stdio process.
This matters because Claude Code and Claude Desktop handle MCP transports differently.
\nFor Claude Code, HTTP works great:
\nclaude mcp add -s user -t http google-ads-mcp \"http://127.0.0.1:8000/mcp\"\nBut you need the server actually running first. I installed it with pipx:
\npipx install \"git+https://github.com/google-marketing-solutions/google_ads_mcp.git\"\nThen created a launchd service so it starts automatically and survives reboots:
\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\"\n \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n<plist version=\"1.0\">\n<dict>\n <key>Label</key>\n <string>com.google-ads-mcp</string>\n <key>ProgramArguments</key>\n <array>\n <string>/opt/homebrew/bin/pipx</string>\n <string>run</string>\n <string>--spec</string>\n <string>git+https://github.com/google-marketing-solutions/google_ads_mcp.git</string>\n <string>run-mcp-server</string>\n </array>\n <key>EnvironmentVariables</key>\n <dict>\n <key>GOOGLE_ADS_CREDENTIALS</key>\n <string>/Users/yourusername/.google-ads.yaml</string>\n <key>PATH</key>\n <string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin</string>\n </dict>\n <key>RunAtLoad</key>\n <true/>\n <key>KeepAlive</key>\n <true/>\n <key>StandardOutPath</key>\n <string>/tmp/google-ads-mcp.log</string>\n <key>StandardErrorPath</key>\n <string>/tmp/google-ads-mcp.err</string>\n</dict>\n</plist>\nSave that to ~/Library/LaunchAgents/com.google-ads-mcp.plist and load it:
launchctl load ~/Library/LaunchAgents/com.google-ads-mcp.plist\nThe server starts on boot, restarts if it crashes, and logs to /tmp/. Verify it's running:
claude mcp list\n# google-ads-mcp: http://127.0.0.1:8000/mcp (HTTP) - ✓ Connected\nHere's the plot twist. Claude Desktop doesn't support streamable-http transport. It only speaks stdio (and sse for some servers). When you try to point it at an HTTP URL, it shows you a polite error dialog and refuses to load the server.
The Google Ads MCP server hardcodes streamable-http in its source:
mcp_server.run(\n transport=\"streamable-http\",\n show_banner=False,\n)\nNo CLI flag to change it. No environment variable. Just hardcoded.
\nThe fix: a tiny wrapper script that imports the same MCP server object but runs it in stdio mode. Since I already installed the package with pipx, I can use the venv's Python interpreter directly:
\n#!/Users/yourusername/.local/pipx/venvs/google-ads-mcp/bin/python3\n\"\"\"Run Google Ads MCP server in stdio mode for Claude Desktop.\"\"\"\nimport asyncio\nimport os\n\nos.environ.setdefault(\n \"GOOGLE_ADS_CREDENTIALS\",\n os.path.expanduser(\"~/.google-ads.yaml\")\n)\n\nfrom ads_mcp.coordinator import mcp_server\nfrom ads_mcp.scripts.generate_views import update_views_yaml\nfrom ads_mcp.tools import api\n\ndef main():\n asyncio.run(update_views_yaml())\n api.get_ads_client()\n mcp_server.run(\n transport=\"stdio\",\n show_banner=False,\n )\n\nif __name__ == \"__main__\":\n main()\nSave it to ~/.local/bin/google-ads-mcp-stdio, make it executable:
chmod +x ~/.local/bin/google-ads-mcp-stdio\nThen in Claude Desktop's config (~/Library/Application Support/Claude/claude_desktop_config.json):
{\n \"mcpServers\": {\n \"google-ads-mcp\": {\n \"command\": \"/Users/yourusername/.local/bin/google-ads-mcp-stdio\",\n \"args\": []\n }\n }\n}\nRelaunch Claude Desktop. No error dialog. The MCP tools show up in the chat input.
\n┌─────────────────────────────────────────────────────┐\n│ ~/.google-ads.yaml │\n│ (shared credentials file) │\n└──────────────────────┬──────────────────────────────┘\n │\n ┌────────────┴────────────┐\n │ │\n ┌──────▼──────┐ ┌──────▼──────┐\n │ HTTP Server │ │ stdio wrapper│\n │ port 8000 │ │ script │\n │ (launchd) │ │ (on demand) │\n └──────┬──────┘ └──────┬──────┘\n │ │\n ┌──────▼──────┐ ┌──────▼──────┐\n │ Claude Code │ │Claude Desktop│\n │ (dev work) │ │ (marketing) │\n └─────────────┘ └─────────────┘\nTwo clients, two transports, same underlying MCP code, same credentials. Claude Code connects over HTTP to a persistent background server. Claude Desktop spawns its own stdio process on launch.
\nThey don't share a server process, but they don't need to. There's no shared state. Both just authenticate against the Google Ads API independently.
\nThe point isn't just \"I connected a thing to another thing.\" It's about putting the right tools in the right context.
\nWhen I'm in Claude Code working on the landing page, I can ask \"what search terms triggered clicks this week?\" and cross-reference that with PostHog funnel data without leaving my terminal. The code changes I make are informed by real campaign data, not a tab I glanced at an hour ago.
\nWhen I'm in Claude Desktop doing marketing work, I can pull campaign reports, compare ad group performance, and brainstorm copy changes in a conversational flow. No spreadsheet intermediary.
\nSame data source. Different workflows. That's the whole idea behind MCP: you connect your tools once and use them wherever the context calls for it.
\nA few things that will save you time:
\nuse_proto_plus: True is required in your google-ads.yaml. The server crashes with a confusing ValueError without it. None of the setup guides mention this.
The transport is hardcoded. The Google Ads MCP server only runs in streamable-http mode. Claude Desktop only supports stdio. The wrapper script bridges this gap.
Developer token access levels matter. A test account token can only query test accounts. If you need real campaign data, you'll need to apply for Basic Access through Google's API Center. The approval can take a few days.
\nlaunchd vs. manual process. Don't just background the server with &. It won't survive a reboot, and if it crashes at 2 AM, it stays crashed. The launchd plist with KeepAlive: true handles both.
pipx caching. If you use pipx run instead of pipx install, it creates cached venvs that can disappear. For a persistent service, install it properly.
For now, the plumbing is done. Both clients are connected, the server starts on boot, and I can ask Claude about my Google Ads account from anywhere. The hard part, as always, was getting the credentials right and figuring out that two different Claude clients need two different transport protocols for the same MCP server. Nobody warns you about that one.
\nI used this prompt to generate featured image. A split-screen digital illustration showing two computer interfaces connected to a central glowing node. On the left, a dark terminal/CLI environment (Claude Code) with green text. On the right, a clean chat interface (Claude Desktop) with a modern UI. Both connect via glowing data streams to a central hub labeled with the Google Ads logo. The aesthetic is technical but approachable, with a warm color palette of blues, greens, and Google's signature colors. Isometric perspective, clean lines, subtle circuit board patterns in the background. Homelab/developer workspace vibe.
\n","comment_id":"69c828461340aa0001a5047e","feature_image":"https://emir.fyi/content/images/2026/03/f9f1049d-ee22-4549-8afa-3bb77726828a.png","featured":false,"visibility":"public","created_at":"2026-03-28T15:13:10.000-04:00","updated_at":"2026-04-01T10:00:40.000-04:00","published_at":"2026-04-01T10:00:40.000-04:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/giving-claude-desktop-and-claude-code-my-google-ads-account-on-purpose/","excerpt":"The actual problem\n\n\nI have a landing page (I am helping my wife generate leads for her business). It has a Google Ads campaign driving traffic to it. It also has PostHog tracking what visitors do once they arrive. Two systems, two dashboards, two browser tabs I keep open and alt-tab between when I'm trying to figure out if the campaign is actually working.\n\n\nThe workflow looks like this: check Google Ads to see which keywords are spending money, switch to PostHog to see if those visitors are ac","reading_time":6,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"69b37b6ce62aae000136c0da","uuid":"77b08416-9ae4-4305-b330-252efcbdf7e2","title":"I Sat Down to Build an HA Kubernetes Cluster. I Did Not Build an HA Kubernetes Cluster.","slug":"i-sat-down-to-build-an-ha-kubernetes-cluster-i-did-not-build-an-ha-kubernetes-cluster","html":"I sat down tonight to build an HA Kubernetes cluster. I had two BeeLink mini PCs collecting dust, freshly installed with Ubuntu 24.04, both connected to my network over WiFi. Simple plan: switch them to wired, assign static IPs, and start building.
\nTLDR: I did not build an HA Kubernetes cluster tonight.
\nI've been running a single control plane K8s cluster with two worker nodes on Proxmox VMs for a while now. It works, but it's all on one physical host. If Proxmox goes down, everything goes down. Not exactly \"high availability.\" And I have real reasons to care about uptime:
\nWhat actually pushed me to do this now rather than \"eventually\" was hardware failure. One of the ECC RAM slots on my Proxmox server started acting up, the machine restarted on me a few times out of nowhere. Re-seating the DIMMs seems to have fixed it for now, and I've ordered a replacement stick, but \"seems to have fixed it\" isn't exactly confidence-inspiring. I want my services to survive the next time that machine decides to take an unscheduled nap.
\nThe BeeLink mini PCs were perfect for this. Small, quiet, low power, and already sitting in a drawer. I installed Ubuntu 24.04 on both using WiFi during setup, gave them hostnames n1 and n2 (node 1, node 2. Very creative, I know), mounted them in my server rack, and ran ethernet from the patch panel. The plan was straightforward:
\nI got through exactly one and a half of those steps before things went sideways.
\nI started with n2. SSHed in over WiFi, saw the wired interface was DOWN with qdisc noop (kernel-speak for \"I haven't even initialized this thing\"), and in my infinite wisdom decided to do it all in one shot. Replace the entire WiFi netplan config with wired-only and netplan apply.
The SSH session hung. WiFi went down immediately. The wired interface? Also down. On these BeeLinks, the Realtek NIC exists and the driver loads, but it won't auto-negotiate a link until something explicitly brings it up. Netplan killed WiFi before the wired interface had a chance to come alive. No network, no SSH, no way back in.
\nThe machine was sitting headless in a closet. No monitor, no keyboard. Just a blinking power LED and regret.
\nA normal person would hook up a monitor, keyboard, and mouse. A smart person would have a JetKVM or similar KVM over IP for exactly this scenario (JetKVM, if you're reading this, I'm still waiting on that sponsorship). I am neither of these people. I held the power button, waited, pressed it again, and hoped. It worked. pfSense showed a new DHCP lease on the wired MAC. Got lucky. If the config had been bad or the cable loose, I'd be dragging a monitor across the house like a normal person after all.
\nWith n1, I did it properly. The key insight: never remove your working network connection before verifying the replacement works. Obvious in retrospect.
\nThe approach: add wired alongside WiFi first, use netplan try (which auto-reverts if it breaks things), verify SSH works on the new wired IP, and only then remove WiFi in a second pass.
The transitional netplan config looks like this. Both interfaces active, wired preferred via lower metric:
\nnetwork:\n version: 2\n ethernets:\n enp1s0:\n addresses:\n - 192.168.1.1.40/24\n routes:\n - to: default\n via: 192.168.1.1\n metric: 100\n nameservers:\n addresses:\n - 192.168.1.1\n wifis:\n wlo1:\n dhcp4: true\n access-points:\n \"MyWiFiNetwork-real-network-totally\":\n auth:\n key-management: \"psk\"\n password: \"myrealpassword\"\nThe magic command is netplan try --timeout 30 instead of netplan apply. It applies the config but automatically reverts after 30 seconds unless you confirm. If it breaks your connection, you don't have to go find a monitor. It rolls back on its own.
After verifying SSH worked on the wired IP, I removed the wifis block, ran netplan generate to validate, then netplan apply. WiFi went down, wired stayed up. No drama.
| Host | \nIP | \nInterface | \nStatus | \n
|---|---|---|---|
| n1.localdomain | \n192.168.1.40 | \nenp1s0 (wired) | \nStatic, WiFi disabled | \n
| n2.localdomain | \n192.168.1.41 | \nenp1s0 (wired) | \nStatic, WiFi disabled | \n
Both machines are on static IPs, wired only, reachable via .localdomain hostnames (set via my DNS server). The networking took the entire evening.
The actual lesson here isn't about netplan syntax or Realtek drivers. It's about making changes to remote systems you can't physically access.
\nNever remove your working connection before verifying the replacement. Add the new thing alongside the old thing. Test it. Confirm it. Then remove the old thing. This applies to network interfaces, DNS servers, firewall rules. Any change where a mistake means you're locked out. The best part? I already knew all of this. But it's a homelab, it's a local network, and I was trying to move fast. The best part? I already knew all of this. But it's a homelab, it's a local network, and I was trying to move fast. The mental math was simple. If it works, I save five minutes and start building the cluster tonight. If it doesn't, I'm configuring HA Kubernetes some other night. Seemed like a worthwhile bet at the time. I would not take it again.
\nnetplan try exists specifically for this. Use it. It's the difference between \"let me verify this works\" and \"let me go find an HDMI cable.\"
The HA Kubernetes cluster will have to wait for another night. But at least now when I sit down to build it, both machines will be reachable over a wire, with static IPs like civilized infrastructure.
\n","comment_id":"69b37b6ce62aae000136c0da","feature_image":"https://emir.fyi/content/images/2026/03/ChatGPT-Image-Mar-12--2026--11_22_20-PM.png","featured":false,"visibility":"public","created_at":"2026-03-12T22:50:20.000-04:00","updated_at":"2026-03-25T09:47:02.000-04:00","published_at":"2026-03-25T09:47:02.000-04:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/i-sat-down-to-build-an-ha-kubernetes-cluster-i-did-not-build-an-ha-kubernetes-cluster/","excerpt":"I sat down tonight to build an HA Kubernetes cluster. I had two BeeLink mini PCs collecting dust, freshly installed with Ubuntu 24.04, both connected to my network over WiFi. Simple plan: switch them to wired, assign static IPs, and start building.\n\n\nTLDR: I did not build an HA Kubernetes cluster tonight.\n\n\n\nThe Setup\n\n\nI've been running a single control plane K8s cluster with two worker nodes on Proxmox VMs for a while now. It works, but it's all on one physical host. If Proxmox goes down, ever","reading_time":4,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"69b96c111340aa0001a5045f","uuid":"42a01dc7-1b6b-4832-8419-e0abf756a7b2","title":"Adding Self-Hosted Comments to My Ghost Blog with Isso","slug":"adding-self-hosted-comments-to-my-ghost-blog-with-isso","html":"I write blog posts about my homelab projects. People read them, some of them find them useful, and I know this because occasionally someone messages me on Linkedin. But there's no way for readers to leave feedback right there on the post. No \"hey, I tried this and it worked\" or \"you missed a step here\" or \"this saved me three hours.\" That kind of feedback loop makes the posts better for everyone who comes after.
Ghost has built-in comments, but they require readers to create an account first. For a personal tech blog, that's a dealbreaker. Nobody is going to sign up for yet another account just to say \"thanks, this helped.\" I needed something where you type your name, type your comment, and hit submit.
And lastly, because I can hehe. This is what I do I host my own services.
I spent some time looking at what's out there for comment systems that don't require reader accounts. Here's what I found:
\n\n| System | \nFree? | \nAds? | \nSelf-hosted? | \nGuest comments? | \n
|---|---|---|---|---|
| Disqus | \nFree tier has ads | \nYes | \nNo | \nYes | \n
| giscus | \nYes | \nNo | \nNo (GitHub hosted) | \nNo (needs GitHub account) | \n
| Cusdis | \nYes | \nNo | \nYes (Docker) | \nYes | \n
| Isso | \nYes | \nNo | \nYes (Docker) | \nYes | \n
Disqus was out immediately. The free tier injects ads into your comment section, and the paid plans start at $11/month. I didn't escape cloud camera subscriptions just to pay for cloud comment subscriptions.
giscus is clever. It uses GitHub Discussions as the backend, so comments are stored in your repo's discussion threads. Clean UI, reactions, threading. But readers need a GitHub account to comment. Great for a developer tools blog, not great for a general audience.
Cusdis is the minimalist option. Self-hostable, no account required, just nickname and comment. But it's almost too minimal. No markdown, no voting, no threading, no edit/delete. It felt like a textarea with a submit button.
Isso hit the sweet spot. Self-hosted via Docker, anonymous comments with just a name, markdown support, upvote/downvote, threaded replies, and users can edit or delete their own comments within 15 minutes. It also has a built-in admin panel for moderation and uses SQLite, so there's no extra database to manage.
My blog runs on Ghost at emir.fyi, served through Cloudflare -> cloudflared tunnel -> Caddy (HA load balancer) -> Ghost on Portainer (192.168.1.2). I needed to fit Isso into this without adding a new subdomain, new DNS records, new TLS certificates, or new Cloudflare tunnel configs.
The solution: serve Isso under a path on the same domain. Requests to emir.fyi/isso/* go to the Isso container, everything else still goes to Ghost. One Caddy rule, zero new infrastructure.
Following the same pattern as my other Portainer services:
services:\n isso:\n image: ghcr.io/isso-comments/isso:release\n container_name: isso\n restart: unless-stopped\n ports:\n - \"8080:8080\"\n volumes:\n - /opt/isso/config:/config\n - /opt/isso/db:/db\n environment:\n - TZ=America/New_York\n healthcheck:\n test: [\"CMD-SHELL\", \"wget -q -O /dev/null http://localhost:8080/info 2>&1 || exit 1\"]\n interval: 30s\n timeout: 10s\n retries: 3\n logging:\n driver: json-file\n options:\n max-size: \"10m\"\n max-file: \"3\"\nNothing fancy. SQLite database and config mounted as bind mounts to /opt/isso/ on the host. The release tag pins to the latest stable version.
[general]\ndbpath = /db/comments.db\nhost =\n https://emir.fyi\nmax-age = 15m\ngravatar = true\n\n[server]\nlisten = http://0.0.0.0:8080\npublic-endpoint = https://emir.fyi/isso\n\n[moderation]\nenabled = false\npurge-after = 30d\n\n[guard]\nenabled = true\nratelimit = 2\ndirect-reply = 3\nreply-to-self = false\nrequire-author = true\nrequire-email = false\n\n[markup]\noptions = strikethrough, superscript, autolink\n\n[admin]\nenabled = true\npassword = your_strong_password_here\nA few things worth noting:
host must match your blog's URL exactly. Isso uses this for CORS, so if it doesn't match, comments will fail silently.public-endpoint tells Isso where it lives from the outside. This is the path that gets embedded in the comment widget.moderation = false because I want comments to post immediately. Life's too short to moderate a personal blog.guard is still on. Rate limiting (2 comments per minute per IP) and requiring an author name keeps the low-effort spam out.require-email = false because I don't want to create friction. Name and comment, that's it.The only infrastructure change was adding a path handler to the existing :80 block in the Caddyfile. This is the block that Cloudflare's tunnel hits for emir.fyi:
:80 {\n handle_path /isso/* {\n reverse_proxy 192.168.1.2:8080\n }\n\n handle {\n reverse_proxy 192.168.1.2:2368 {\n header_up X-Forwarded-Proto https\n }\n }\n}\nhandle_path strips the /isso/ prefix before forwarding to the Isso container, so Isso sees clean paths. Everything else falls through to Ghost. Deployed via the existing Ansible playbook across all three HA load balancer nodes.
The last piece is embedding the Isso widget in blog posts. I edited the Attila theme's post.hbs to include the Isso script:
<section class=\"post-comments\">\n <div id=\"isso-thread\"></div>\n <script data-isso=\"https://emir.fyi/isso/\"\n src=\"https://emir.fyi/isso/js/embed.min.js\"></script>\n</section>\nThis goes right before the existing {{#if comments}} block. Ghost needs a restart to pick up the theme change.
Isso's default styles assume a white background, which looks fine on light mode but terrible on dark mode. White input fields and invisible text on a dark page. Since my theme (Attila) toggles dark mode via a theme-dark class on the HTML element, I added CSS through Ghost's code injection (Settings -> Code injection -> Site Header):
.theme-dark #isso-thread .isso-postbox .isso-textarea,\n.theme-dark #isso-thread .isso-postbox input[type=\"text\"] {\n background-color: #2a2a2a;\n color: #e0e0e0;\n border-color: #555;\n}\n.theme-dark #isso-thread .isso-postbox input[type=\"submit\"],\n.theme-dark #isso-thread .isso-postbox input[name=\"preview\"] {\n background-color: #444;\n color: #e0e0e0;\n border-color: #555;\n}\n.theme-dark #isso-thread .isso-comment .isso-text,\n.theme-dark #isso-thread .isso-comment .isso-comment-header {\n color: #e0e0e0;\n}\n.theme-dark #isso-thread h4 {\n color: #ccc;\n}\n.theme-dark #isso-thread a {\n color: #6cb4ff;\n}\nThat last rule is important. Isso renders Edit/Delete links in default blue, which is invisible on a dark background. Light blue (#6cb4ff) fixes that.
Isso has an optional Website field where commenters can link their site. The problem is that Isso's server validates it strictly: if you type example.com without https://, it returns a 400 Bad Request with zero feedback in the UI. The comment just doesn't post and you have no idea why.
I couldn't change Isso's server-side validation, so I added client-side validation via Ghost's code injection (Site Footer) that highlights the field and updates the label when the URL format is wrong:
document.addEventListener('DOMContentLoaded', function() {\n var observer = new MutationObserver(function() {\n var website = document.querySelector('#isso-postbox-website');\n if (website && !website.dataset.validated) {\n website.dataset.validated = 'true';\n var label = document.querySelector('label[for=\"isso-postbox-website\"]');\n var originalText = label.textContent;\n\n website.addEventListener('input', function() {\n var val = website.value.trim();\n if (val === '' || /^https?:\\/\\/.+/.test(val)) {\n website.style.borderColor = '';\n label.textContent = originalText;\n } else {\n website.style.borderColor = 'red';\n label.textContent = originalText + ' - invalid';\n }\n });\n }\n });\n observer.observe(document.body, {childList: true, subtree: true});\n});\nA MutationObserver because Isso's widget loads asynchronously into an iframe/div after the page renders. The validation fires on every keystroke: empty is fine (it's optional), anything starting with http:// or https:// is fine, anything else turns the border red and appends \" - invalid\" to the label. The form still submits either way, but at least the user can see something is wrong before they wonder why their comment disappeared.
Since Isso is publicly accessible at emir.fyi/isso/, I thought about what's exposed:
/isso/admin/ is password-protected with a strong password. No rate limiting on login attempts though, so the password matters.emir.fyi only. You can't post comments from a different origin.The container itself is minimal. It runs as an unprivileged user, stores data in SQLite (no exposed database ports), and the only network exposure is port 8080 proxied through Caddy.
Comment form at the bottom of every post. Type your name, optionally your email (for Gravatar only, not displayed), optionally a website, write your comment in markdown, hit Submit. It posts immediately. You get 15 minutes to edit or delete your own comment. Upvote and downvote. Threaded replies.
No account creation. No OAuth flow. No \"sign in with Google.\" Just a comment box that works.
If I were starting fresh, I'd look harder at whether Isso is still actively maintained. The last release was 0.13.0 and the GitHub activity is sporadic. It works well today, but I'm mentally prepared to swap it out if it stops getting updates. The data is in a single SQLite file, so migration would be straightforward.
I'd also consider running Isso behind its own subdomain from the start if I had wildcard DNS and certs already set up. The path-based routing works fine, but a subdomain would be cleaner and wouldn't require the handle_path prefix stripping in Caddy.
But for a weekend project that adds real community interaction to a blog? Isso does exactly what I needed. No SaaS dependency, no ads, no reader accounts, and it runs on infrastructure I already had.
","comment_id":"69b96c111340aa0001a5045f","feature_image":"https://emir.fyi/content/images/2026/03/ChatGPT-Image-Mar-17--2026--11_10_24-AM.png","featured":false,"visibility":"public","created_at":"2026-03-17T10:58:25.000-04:00","updated_at":"2026-03-22T11:16:46.000-04:00","published_at":"2026-03-22T11:16:46.000-04:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/adding-self-hosted-comments-to-my-ghost-blog-with-isso/","excerpt":"Why I wanted comments\n\nI write blog posts about my homelab projects. People read them, some of them find them useful, and I know this because occasionally someone messages me on Linkedin. But there's no way for readers to leave feedback right there on the post. No \"hey, I tried this and it worked\" or \"you missed a step here\" or \"this saved me three hours.\" That kind of feedback loop makes the posts better for everyone who comes after.\n\nGhost has built-in comments, but they require readers to cre","reading_time":6,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"69b95967cbda360001756cb1","uuid":"9751da81-9076-422c-a63a-a2ae7559ebae","title":"Frigate NVR 0.17 on Apple Silicon: Running AI Detection on the M1 Neural Engine","slug":"frigate-0-17-on-apple-silicon-running-ai-detection-on-the-m1-neural-engine","html":"When I first set up Frigate NVR on my M1 Mac Mini, the AI detection worked, but it was running entirely on the CPU. Three Amcrest PoE cameras, each pushing a detect stream, with every frame getting processed through a CPU-based ONNX detector at ~29ms per inference. The M1 handled it, but the machine was running hot, skipping frames, and the recording pipeline kept backing up with warnings every five seconds about \"too many unprocessed recording segments.\" The M1 has this beautiful 16-core Neural Engine sitting right there, purpose-built for exactly this kind of workload, and I wasn't using it at all.
Frigate 0.17 changed that. The release introduced native Apple Silicon support through a ZMQ proxy architecture that lets the Neural Engine handle object detection from outside the Docker container. I had to try it.
Here's the catch that makes this interesting: Docker Desktop on macOS runs a Linux VM under the hood. Frigate lives inside that VM, and from inside a Linux container, you simply cannot access Apple's CoreML framework or the Neural Engine. It doesn't matter how much RAM you give Docker or how many CPU cores you allocate. The NPU is invisible to the container.
Before I figured this out, I actually thought the problem was resource allocation. I had Docker set to 12GB of RAM on a 16GB machine (leaving macOS gasping for air with 193MB free and 6.5GB in the compressor). I was hoping giving it more memory would somehow offset work to Apple's neural engine. Reducing Docker to 8GB helped the system breathe, but detection was still running on CPU.
The architecture is clever. Instead of trying to access the Neural Engine from inside the container, Frigate 0.17 supports a ZMQ detector that runs as a separate process on the Mac host. The flow looks like this:
The latency added by this hop is negligible since both processes run on the same machine.
standard-arm64 Docker imageMy M1 only had Python 3.9.6 from Xcode Command Line Tools. The detector needs 3.11+.
curl https://pyenv.run | bash\nAdd to your ~/.zshrc:
export PYENV_ROOT=\"$HOME/.pyenv\"\n[[ -d $PYENV_ROOT/bin ]] && export PATH=\"$PYENV_ROOT/bin:$PATH\"\neval \"$(pyenv init -)\"\nThen install Python 3.11:
pyenv install 3.11\ncd ~\ngit clone https://github.com/frigate-nvr/apple-silicon-detector.git\ncd apple-silicon-detector\npyenv local 3.11\nmake install\nThe detector needs an ONNX model. Frigate doesn't ship one for Apple Silicon, so you need to export it yourself. I used ultralytics to grab YOLOv9t (the tiny variant, perfect for the M1) and export it at 320x320:
# Inside the apple-silicon-detector venv\nvenv/bin/pip3 install ultralytics onnx onnxslim\nvenv/bin/python3 -c \"\nfrom ultralytics import YOLO\nmodel = YOLO('yolov9t.pt')\nmodel.export(format='onnx', imgsz=320)\n\"\nThen copy it where Frigate can find it:
mkdir -p ~/frigate/config/model_cache\ncp yolov9t.onnx ~/frigate/config/model_cache/yolo.onnx\nNote: If your pyenv Python was built without lzma support (common if you don't have xz installed), you'll hit a ModuleNotFoundError: No module named '_lzma' from torchvision. You can work around this by mocking the lzma module before importing ultralytics, or just install xz before building Python (brew install xz && pyenv install 3.11).
Update your config.yml to use the ZMQ detector and the YOLO model:
detectors:\n apple-silicon:\n type: zmq\n endpoint: tcp://host.docker.internal:5555\n\nmodel:\n model_type: yolo-generic\n width: 320\n height: 320\n input_tensor: nchw\n input_dtype: float\n path: /config/model_cache/yolo.onnx\n labelmap_path: /labelmap/coco-80.txt\nThe host.docker.internal hostname is how Docker containers reach the Mac host. Port 5555 is where the ZMQ detector listens.
I ran the detector with nohup for testing, but it wouldn't survive a reboot. And I created a launchd service:
cat > ~/Library/LaunchAgents/com.frigate.apple-silicon-detector.plist << 'EOF'\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\"\n \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n<plist version=\"1.0\">\n<dict>\n <key>Label</key>\n <string>com.frigate.apple-silicon-detector</string>\n <key>ProgramArguments</key>\n <array>\n <string>/path/to/apple-silicon-detector/venv/bin/python3</string>\n <string>/path/to/apple-silicon-detector/detector/zmq_onnx_client.py</string>\n <string>--endpoint</string>\n <string>tcp://*:5555</string>\n <string>--providers</string>\n <string>CoreMLExecutionProvider</string>\n <string>CPUExecutionProvider</string>\n </array>\n <key>WorkingDirectory</key>\n <string>/path/to/apple-silicon-detector</string>\n <key>RunAtLoad</key>\n <true/>\n <key>KeepAlive</key>\n <true/>\n <key>StandardOutPath</key>\n <string>/path/to/apple-silicon-detector/detector.log</string>\n <key>StandardErrorPath</key>\n <string>/path/to/apple-silicon-detector/detector.err</string>\n</dict>\n</plist>\nEOF\nLoad it:
launchctl load ~/Library/LaunchAgents/com.frigate.apple-silicon-detector.plist\nThe KeepAlive key means launchd will restart the detector if it crashes. RunAtLoad means it starts automatically on login. Between this and Docker Desktop's own auto-start, the whole stack comes up on boot without touching anything.
Start the detector first (or let launchd handle it), then start Frigate:
docker start frigate\nCheck the detector log to confirm CoreML loaded:
INFO - Loading ONNX model with providers: ['CoreMLExecutionProvider']\nINFO - Model input: images, shape: [1, 3, 320, 320], type: tensor(float)\nINFO - Model ready for inference\nI let it run overnight with three cameras. Here's what changed:
\n\n| Metric | \nCPU Detector (before) | \nNeural Engine (after) | \n
|---|---|---|
| Inference speed | \n28.8ms | \n17.1ms | \n
| Skipped frames | \n91-109 fps skipped | \n0 | \n
| Recording warnings | \nEvery 5 seconds | \nNone | \n
| Detection CPU usage | \n6% (detector process) | \n0% (offloaded to NPU) | \n
| Camera | \nStream FPS | \nDetection FPS | \nSkipped | \n
|---|---|---|---|
| Camera 1 | \n5.0 | \n15.4 | \n0 | \n
| Camera 2 | \n5.0 | \n28.5 | \n0 | \n
| Camera 3 | \n5.1 | \n8.0 | \n0 | \n
Total detection throughput: 51.9 FPS across all cameras. Zero frames dropped. The \"too many unprocessed recording segments\" warnings that plagued the CPU detector? Completely gone.
The overall system still runs warm (Frigate container uses ~94% of its allocated 4 CPU cores), but that's ffmpeg doing stream decoding and recording, not detection. The actual AI inference is invisible from a CPU perspective because it's happening on dedicated silicon.
The M1's Neural Engine is genuinely underutilized in most homelab setups. Apple doesn't make it easy to access from non-native environments (hence the Docker workaround), but the Frigate team built an elegant solution with the ZMQ proxy pattern. The key insight is that you don't need the detection to happen inside the container. Shipping frames over a local TCP socket adds almost no latency, and you get to use hardware that would otherwise sit completely idle.
If you're running Frigate on Apple Silicon with the CPU detector, this upgrade is worth the 30 minutes it takes to set up. The M1 Mac Mini went from struggling with three cameras to handling them effortlessly, and I have headroom to add more without worrying about the detection pipeline falling behind.
The Mac Mini M1 continues to be one of the best value machines in my homelab. NVR with AI detection, monitoring hub, and it does it all silently with no fan noise and barely any power draw. Not bad for a machine Apple doesn't even sell anymore.
","comment_id":"69b95967cbda360001756cb1","feature_image":"https://emir.fyi/content/images/2026/03/ChatGPT-Image-Mar-17--2026--09_56_44-AM.png","featured":false,"visibility":"public","created_at":"2026-03-17T09:38:47.000-04:00","updated_at":"2026-03-18T10:08:34.000-04:00","published_at":"2026-03-18T10:08:34.000-04:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/frigate-0-17-on-apple-silicon-running-ai-detection-on-the-m1-neural-engine/","excerpt":"Background\n\nWhen I first set up Frigate NVR on my M1 Mac Mini, the AI detection worked, but it was running entirely on the CPU. Three Amcrest PoE cameras, each pushing a detect stream, with every frame getting processed through a CPU-based ONNX detector at ~29ms per inference. The M1 handled it, but the machine was running hot, skipping frames, and the recording pipeline kept backing up with warnings every five seconds about \"too many unprocessed recording segments.\" The M1 has this beautiful 16","reading_time":5,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"69b546d0cbda360001756c6b","uuid":"f803b566-c80b-42c7-826f-8de5ead30955","title":"My Blog Told Me It Was Vulnerable. It Was Right.","slug":"my-blog-told-me-it-was-vulnerable-it-was-right","html":"I self host my blog. The whole thing, Ghost CMS, MySQL database, all of it, runs on a Docker host in my house. A Cloudflare tunnel punches it out to the internet so you can read posts at emir.fyi without my home IP being exposed. It sits alongside a dozen other self hosted services on a machine I manage through Portainer: a note taking app, a document management system, a container registry, monitoring agents. The blog is the only one that faces the public internet.
\nWhy self host a blog in 2026? Honestly, because I can. I already have the infrastructure for my homelab, and running Ghost on hardware I control means no monthly bills, no platform risk, no \"we're pivoting to AI and shutting down the blogging product\" emails. Plus, I write about self hosting. It would be a little embarrassing to do that from a managed WordPress instance.
\nThe setup had been running fine for some time now. Ghost 6.14.0 on port 2368, MySQL 8.4 backing it, both as Docker containers on a shared network. Traffic flows in through Cloudflare, hits a Caddy reverse proxy on my HA load balancer, and lands on Ghost. Simple, stable, boring in the best way.
\nThen I opened my Ghost admin panel and saw the kind of message that makes you put down your coffee:
\n\n\n\"Update Ghost now: your Ghost site is vulnerable to an attack that lets unauthenticated attackers read arbitrary data from the database.\"
\n
Not \"hey, there's a minor update available.\" Not \"consider upgrading when convenient.\" Unauthenticated attackers can read your entire database. Cool. Love that for me.
\nThis is the part of self hosting nobody puts in the brochure. When you run your own stuff, you're the one who gets paged. There's no managed hosting provider absorbing the hit and pushing a patch while you sleep. The CVE lands, the banner appears, and it's on you.
\nCVE-2026-26980 is a SQL injection in Ghost's Content API. The Content API is the public facing, no auth required API that serves your blog posts to the world. Somebody figured out you could inject SQL through it and read whatever you want from the database. User credentials, session tokens, email addresses, draft posts, everything.
\nAffected versions: Ghost 3.24.0 through 6.19.0. I was running 6.14.0. My blog is publicly accessible via Cloudflare tunnel, so this wasn't a theoretical risk. Anyone on the internet could have been reading my database like a book.
\nCVSS score: 9.4. That's \"stop what you're doing and fix this now\" territory.
\nSimple enough:
\nThat third one is what turned a 10 minute patch into an evening project.
\nWhen I went to update Ghost, I discovered something embarrassing. Every other service on my Portainer host (SiYuan, Paperless, Redis, Errbit etc.) had proper docker-compose.yml files. Source of truth in the homelab repo, deployed copies on the host, secrets in .env files. Clean. Reproducible. Professional, even.
Ghost? The blog that's publicly accessible on the internet? The one service that actually has a CVE against it? Manually created through the Portainer UI. No compose file. No IaC. Just vibes and a container that's been running since whenever I clicked \"deploy\" long time ago.
\nI had to docker inspect the container just to figure out what environment variables it was using. That's how you know your infrastructure management has gaps.
First priority: get the vulnerable version off the internet. The actual patching process was straightforward since the data lives in Docker volumes that persist independently of the container.
\nHere's the approach:
\nThe key thing to understand is that Ghost stores everything in two places. A ghost-content Docker volume holds themes, images, and settings. A MySQL database holds everything else. Both are external to the container. Swapping the container image is like changing the engine in a car while keeping all the cargo. Nothing gets lost.
Since the SQLi vulnerability could read arbitrary database data, I had to assume the MySQL passwords were compromised. That means rotating both the ghost database user and the root MySQL password.
You change passwords in a running MySQL container like this:
\nALTER USER 'ghost'@'%' IDENTIFIED BY 'new-ghost-password-here';\nALTER USER 'root'@'localhost' IDENTIFIED BY 'new-root-password-here';\nFLUSH PRIVILEGES;\nOne gotcha: the MYSQL_ROOT_PASSWORD and MYSQL_PASSWORD environment variables in a MySQL container are only used on first initialization. Changing them in your compose file doesn't change the actual passwords in the database. You have to change them in MySQL first, then update the compose config to match. If you do it the other way around, your new container will try to connect with the new password and MySQL will say no.
Here's what the Ghost stack looks like properly codified:
\nservices:\n ghost:\n image: ghost:6.21.2\n container_name: emir.fyi-blog\n restart: always\n ports:\n - \"2368:2368\"\n volumes:\n - ghost-content:/var/lib/ghost/content\n env_file: .env\n environment:\n url: https://yourdomain.com\n database__client: mysql\n database__connection__host: mysql-db\n database__connection__user: ghost\n database__connection__database: ghost\n depends_on:\n mysql:\n condition: service_healthy\n healthcheck:\n test: [\"CMD-SHELL\", \"node -e \\\"const h=require('http');h.get('http://localhost:2368/ghost/api/admin/site/',r=>{process.exit(r.statusCode<500?0:1)}).on('error',()=>process.exit(1))\\\"\"]\n interval: 30s\n timeout: 10s\n retries: 3\n\n mysql:\n image: mysql:8.4\n container_name: mysql-db\n restart: always\n ports:\n - \"3306:3306\"\n volumes:\n - mysql-data:/var/lib/mysql\n env_file: .env\n environment:\n MYSQL_DATABASE: ghost\n MYSQL_USER: ghost\n healthcheck:\n test: [\"CMD-SHELL\", \"mysqladmin ping -u root -p\\\"$$MYSQL_ROOT_PASSWORD\\\"\"]\n interval: 30s\n timeout: 10s\n retries: 3\n\nvolumes:\n ghost-content:\n external: true\n mysql-data:\n external: true\nThe .env file (which never gets committed to the repo) holds the secrets:
database__connection__password=your-ghost-db-password\nMYSQL_PASSWORD=your-ghost-db-password\nMYSQL_ROOT_PASSWORD=your-mysql-root-password\nNotice the volumes are marked external: true. That means Docker won't try to create new ones. It'll use the existing volumes with all your data intact. This is what makes the upgrade safe. Your posts, your themes, your images, they're all in those volumes. The container is just the runtime.
The depends_on with condition: service_healthy is important too. Ghost will crash on startup if MySQL isn't ready yet. Without the health dependency, Docker starts both containers simultaneously and Ghost tries to connect to a database that's still initializing. Ask me how I know.
While fixing this, I realized I should document how container management works on Portainer so I don't end up with another rogue manually created container. Here's the standard I landed on:
\nSource of truth: docker/<service>/docker-compose.yml in the homelab Git repo. This is what gets version-controlled, reviewed, and tracked.
Deployed to: /opt/<service>/compose/ on the Portainer host. This is where the running config lives.
Secrets: .env file in the compose directory on the host. A .env.example with placeholder values gets committed to the repo so someone (future me) knows what variables are needed.
Deploy/update: scp the compose file to the host, then cd /opt/<service>/compose && docker compose up -d.
Healthchecks required: Every container gets a healthcheck. No exceptions. Beszel monitors container health, and a container without a healthcheck is invisible to monitoring.
\nIt's simple, it's boring, and it means I'll never have to docker inspect a container to figure out how it was configured again. (I sure hope I don't regret saying never, again. Not gonna happen take 33 I guess)
The technical fix here took maybe 15 minutes. Pull image, rotate credentials, deploy. The part that took longer, and mattered more, was realizing that my most publicly exposed service was the least well managed one.
\nThere's a pattern in homelabs (and honestly, in production environments too) where the first thing you set up gets the least love. You deploy Ghost early on, it works, you move on to shinier projects. Six months later every other service has proper IaC and the blog is held together by a manually created container and muscle memory.
\nCVE-2026-26980 was the kick I needed to fix that. If the vulnerability hadn't forced my hand, Ghost would probably still be running on vibes. Sometimes it takes a 9.4 CVSS score to make you do the housekeeping you've been putting off.
\nThe blog is now running Ghost 6.21.2 with fresh credentials, a proper compose file, and a documented standard for how every container on Portainer should be managed. Not bad for a Saturday morning. Done with all this before my kids woke up.
\n","comment_id":"69b546d0cbda360001756c6b","feature_image":"https://emir.fyi/content/images/2026/03/ChatGPT-Image-Mar-14--2026--07_56_14-AM.png","featured":false,"visibility":"public","created_at":"2026-03-14T07:30:24.000-04:00","updated_at":"2026-03-14T07:56:37.000-04:00","published_at":"2026-03-14T07:52:15.000-04:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/my-blog-told-me-it-was-vulnerable-it-was-right/","excerpt":"I self host my blog. The whole thing, Ghost CMS, MySQL database, all of it, runs on a Docker host in my house. A Cloudflare tunnel punches it out to the internet so you can read posts at emir.fyi without my home IP being exposed. It sits alongside a dozen other self hosted services on a machine I manage through Portainer: a note taking app, a document management system, a container registry, monitoring agents. The blog is the only one that faces the public internet.\n\n\nWhy self host a blog in 202","reading_time":6,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"69b2c54ae62aae000136c0a3","uuid":"9ce0cf93-f541-4395-b75b-554cb20947a4","title":"Running Frigate NVR on M1 Mac Mini with TrueNAS NFS Storage","slug":"running-frigate-nvr-on-m1-mac-mini-with-truenas-nfs-storage","html":"I got tired of changing Blink camera batteries and of the thought that someone in the cloud might have recordings of my family's every move.
\nThat's it. That's the origin story. Every now and then I'd get the notification, trudge outside, swap batteries, and wonder why I was paying a cloud subscription for the privilege of watching grainy clips of my own front yard on someone else's server. So I decided to fix it. Overkill style.
\nI'm a homelabber. I build things to learn things. Half the reason any project gets a green light in my house is because there's something new I get to figure out along the way. This project had a lot of those moments. It was my first time using a PoE switch. It had been a while since I'd fished cable through an attic or soffits, and even longer since I'd worked with PVC conduit outside, cementing it and pulling cable through. Good excuse to dust off those skills. And then Frigate itself was completely new territory for me.
\nThe end goal is straightforward: three wired cameras to start, seven days of continuous recordings, and AI detection for people, cars, and animals. Eventually, when the whole property is covered, I want to figure out who keeps leaving dog poop on my lawn. I haven't worked out what happens after I identify them. Confrontation? A politely worded letter? A wall of shame? That's a problem for future me.
\nThe surveillance software is Frigate, an open source NVR with built-in AI object detection. I'd never run it before, but it looked like one of those cool open source projects that punches way above its weight. It is.
\nI watched NetworkChuck's video where he set up Frigate with Reolink cameras and a Google Coral TPU on a Raspberry Pi, and later scaled it to a gaming PC. Great video. But I had different hardware laying around and different priorities, so my setup diverged pretty quickly:
\n| NetworkChuck's Setup | \nMy Setup | \n
|---|---|
| Raspberry Pi / Gaming PC | \nM1 Mac Mini (collecting dust) | \n
| Google Coral TPU ($100) | \nM1 Neural Engine via Apple Silicon detector (built-in, free) | \n
| Reolink cameras (wireless) | \nAmcrest cameras (wired PoE) | \n
| Local storage / external drive | \nTrueNAS NAS via NFS (22TB) | \n
Going wired was deliberate. Chuck's biggest headache was 10 wireless cameras destroying his WiFi network with RTSP retransmits. His TX retry rate hit 29%. He had to add a third access point, stagger camera reboots, and switch to constant bit rate just to keep things from falling apart. None of that is a problem with Ethernet.
\nThe M1 Mac Mini was just sitting in a drawer. It has a Neural Engine that can handle AI inference without buying a separate accelerator. And the TrueNAS NAS was already running with 22TB of storage, so pointing Frigate at it seemed obvious. It was not obvious. More on that shortly.
\n\n\nGoogling one of these? You're in the right place.
\n\n
docker desktop macOS NFS volume mount NAS \"permission denied\"/docker desktop mac \"host_mnt\" \"operation not permitted\"/frigate NVR macOS Docker Desktop NAS storage/frigate docker compose NFS volume TrueNAS/TrueNAS SCALE NFS \"allow non-root mount\" Docker/frigate NVR Mac mini Apple Silicon M1 setup/frigate amcrest camera RTSP macOS docker/frigate config KeyError ffmpeg disabled camera
This was supposed to be the easy part. I have a TrueNAS box with 22TB of storage sitting on my network. Frigate needs somewhere to store recordings. Just point one at the other, right?
\nI spent more time getting NAS storage working than any other part of this project. If you're running Frigate on Linux, you mount your NFS share and move on with your life. On Docker Desktop for macOS, nothing works the way you expect. What follows is every wall I hit and how I got past it.
\nBoth NFS and SMB can share files over a network, but NFS is the better choice for Frigate:
\nWhen trying to connect Frigate on Docker Desktop (macOS) to NAS storage, there are three approaches. I tried all of them.
\nMount the NFS share on macOS first (sudo mount -t nfs ...), then reference that path in docker-compose volumes. This is the standard approach on Linux.
Does NOT work on Docker Desktop for macOS. Docker Desktop's Linux VM translates host paths to /host_mnt/... and cannot traverse NFS mount points on the host. You'll get operation not permitted errors.
Let Docker's Linux VM mount NFS directly from the NAS, bypassing macOS entirely. Define an NFS volume in docker-compose with driver: local and driver_opts. The VM talks directly to the NAS.
This is what works. Requires enabling \"Allow non-root mount\" on TrueNAS so Docker's VM can connect from non-privileged ports. THIS IS THE MOST IMPORTANT SETTING.
\nSimilar to Option B but using SMB instead of NFS. Docker supports CIFS volumes with type: cifs in driver_opts. Requires username/password configuration and has the performance drawbacks mentioned above. A viable fallback if NFS isn't an option.
/host_mnt path translation bug: This is a known Docker Desktop for Mac issue. Docker Desktop's VM prefixes host paths with /host_mnt/ and cannot handle NFS/network mount points on the host. Switching the file sharing backend from VirtioFS to osxfs has been reported as a workaround but is unreliable./media/frigate directory inside the container and fill up local storage. Using soft in NFS mount options helps Frigate fail gracefully rather than hang. The restart: unless-stopped policy will keep retrying.config.yml to auto-delete old recordings.What follows is every problem I ran into, in order. If you're here from Google, you probably hit one of these exact errors. Welcome. I feel your pain.
\nMy first attempt was the obvious one. Mount the NFS share on macOS, then tell Docker to use that path. Simple. Except:
\nError response from daemon: error while creating mount source path '/host_mnt/Volumes/frigate':\nmkdir /host_mnt/Volumes/frigate: operation not permitted\nWhy: Docker Desktop on macOS runs inside a Linux VM. Host paths get translated to /host_mnt/... inside the VM. NFS mount points on macOS don't pass through this translation layer and the VM can't see or create them.
Solution: Don't mount NFS on macOS at all. Let Docker's Linux VM mount NFS directly from the NAS using Docker's built-in NFS volume driver:
\nvolumes:\n frigate-media:\n driver: local\n driver_opts:\n type: nfs\n o: addr=192.168.1.7,rw,nfsvers=3,nolock,soft\n device: \":/mnt/Storage/frigate\"\nGreat, so Docker needs to mount NFS directly. I set that up, ran docker compose up, and:
failed to mount local volume: mount :/mnt/Storage/frigate:...: permission denied\nWhy: Docker Desktop's Linux VM connects to NFS from a non-privileged port (above 1024). By default, NFS servers reject connections from non-privileged ports, a security holdover from the Unix days where only root could use ports below 1024.
\nSolution: On TrueNAS (tested on v25.04.2.4), enable \"Allow non-root mount\":
\nAt this point I was getting good at reading the words \"Permission denied.\" This time the mount succeeded, but writing to it didn't.
\nWhy: The NFS share's default permissions don't map the connecting user to a user with write access on the NAS filesystem.
\nSolution: On the TrueNAS NFS share settings for the Frigate dataset:
\nrootwheelThis maps all connecting users (including Docker) to root on the NAS, granting full read/write. Acceptable for a dedicated Frigate media folder.
\nNAS finally working. Time to actually run Frigate. It crashed immediately.
\nKeyError: 'ffmpeg'\nWhy: Even disabled cameras in config.yml require an ffmpeg.inputs section. Frigate validates the config structure before checking if the camera is enabled.
Solution: Always include the ffmpeg block, even for placeholder/disabled cameras:
cameras:\n dummy_camera:\n enabled: false\n ffmpeg:\n inputs:\n - path: rtsp://127.0.0.1:8554/dummy\n roles:\n - detect\nThis one was maddening. I could see the camera feed in the Amcrest web UI. I could pull frames with ffmpeg manually. But Frigate kept saying 401 Unauthorized or wrong user/pass. The password was correct. I triple-checked.
Why: RTSP URLs follow standard URL format (rtsp://user:pass@host/path). Characters like ?, @, :, #, & in the password break URL parsing. Both ffmpeg and go2rtc interpret ? as the start of a query string, not part of the password. URL-encoding (%3F) gets double-encoded depending on whether the value is quoted in YAML.
Solution: Change your camera password to avoid special URL characters. Stick to letters and numbers. This is easier than fighting encoding issues across multiple layers (YAML → go2rtc → ffmpeg → RTSP).
\nAvoid in camera passwords: ? @ : # & % /
This one isn't really a problem. It's more of a \"you should do this and here's why.\"
\nWithout go2rtc, every viewer of your camera (Frigate detection, Frigate recording, Home Assistant, your phone) opens a separate RTSP connection directly to the camera. That's four connections to one camera. Multiply by ten cameras and your network is doing unnecessary work. Some cameras don't even handle multiple concurrent RTSP connections well.
\nFrigate already includes go2rtc, a lightweight stream proxy. It connects to the camera once and re-serves the stream locally. All consumers read from rtsp://127.0.0.1:8554/stream_name instead of hitting the camera directly. This is exactly what NetworkChuck did in his video to fix his WiFi issues, and it's even more important if you plan to add Home Assistant later.
Config:
\ngo2rtc:\n streams:\n front_door_sub:\n - rtsp://admin:yourpassword@192.168.1.30:554/cam/realmonitor?channel=1&subtype=1\n\ncameras:\n front_door:\n ffmpeg:\n inputs:\n - path: rtsp://127.0.0.1:8554/front_door_sub\n roles:\n - detect\nThis is especially important if you plan to add Home Assistant or view streams from multiple devices.
\nWith NAS storage sorted and Frigate running, this is where it actually starts feeling like a surveillance system. I unboxed one camera at a time, got it fully working end-to-end before moving to the next. Resist the urge to plug them all in at once. Trust me.
\nping -c 1 192.168.1.255 && arp -a\nnc -z -w 3 192.168.1.X 80 && echo \"HTTP open\" # Web UI\nnc -z -w 3 192.168.1.X 554 && echo \"RTSP open\" # Video stream\nnc -z -w 3 192.168.1.X 37777 && echo \"API open\" # Amcrest API\nhttp://CAMERA_IP and set a password (avoid special URL characters like ?@:#&)The camera should be a dumb video pipe, Frigate handles all the AI.
\nMain stream (high-res, for recording):\nrtsp://admin:PASSWORD@CAMERA_IP:554/cam/realmonitor?channel=1&subtype=0\n\nSubstream (low-res, for AI detection):\nrtsp://admin:PASSWORD@CAMERA_IP:554/cam/realmonitor?channel=1&subtype=1\nEach camera sends two RTSP streams to Frigate via go2rtc:
\nsubtype=0), full resolution (e.g. 2960x1668 or 3840x2160), used for recording. This is the high-quality footage you review.subtype=1), lower resolution (e.g. 704x480), used for AI detection. The AI doesn't need 4K to spot a person, lower res means less CPU/GPU work.go2rtc proxies both streams locally so Frigate (and any other viewer) connects to 127.0.0.1:8554 instead of hitting the camera directly.
Here's where the 22TB NAS earns its keep. I wanted both: continuous recording as a safety net (in case AI misses something), plus longer retention for events that actually matter. Frigate 0.17 supports exactly this with layered retention:
\nrecord:\n enabled: true\n continuous:\n days: 7 # Keep ALL footage for 7 days, then auto-delete\n alerts:\n retain:\n days: 30 # Person/car detections kept 30 days\n detections:\n retain:\n days: 14 # Dog/cat detections kept 14 days\nUpdated March 18, 2026 with real usage data after running three cameras for over a week.
\nThese are real numbers from my setup, not theoretical calculations. After running three cameras for over a week, here's what the NAS is eating per stream:
\n| Stream | \nResolution | \nBitrate | \nPer Hour | \nPer Day | \n7 Days | \n
|---|---|---|---|---|---|
| 4K main | \n3840x2160 | \n~15 Mbps | \n~6.7 GB | \n~64 GB | \n~450 GB | \n
| 5MP main | \n2960x1668 | \n~8 Mbps | \n~3.6 GB | \n~33 GB | \n~230 GB | \n
With all three cameras recording continuously, actual total usage is ~120 GB/day (~40 GB/day per camera). After a full week of 7-day retention, storage settled at ~650 GB on the 22TB NAS. With 22TB available, one could comfortably run 15+ cameras at 7-day continuous retention before storage becomes a concern. This is why I wanted NAS storage instead of a USB drive strapped to the Mac Mini.
\nservices:\n frigate:\n container_name: frigate\n image: ghcr.io/blakeblackshear/frigate:stable-standard-arm64\n restart: unless-stopped\n stop_grace_period: 30s\n shm_size: \"512mb\"\n\n ports:\n - \"8971:8971\"\n - \"8554:8554\"\n - \"8555:8555/tcp\"\n - \"8555:8555/udp\"\n\n volumes:\n - ./config:/config\n - frigate-media:/media/frigate\n - type: tmpfs\n target: /tmp/cache\n tmpfs:\n size: 1000000000\n\n privileged: true\n\nvolumes:\n frigate-media:\n driver: local\n driver_opts:\n type: nfs\n o: addr=192.168.1.7,rw,nfsvers=3,nolock,soft\n device: \":/mnt/Storage/frigate\"\nmqtt:\n enabled: false\n\nobjects:\n track:\n - person\n - dog\n - cat\n - car\n\nrecord:\n enabled: true\n continuous:\n days: 7\n alerts:\n retain:\n days: 30\n detections:\n retain:\n days: 14\n\ngo2rtc:\n streams:\n tree_camera_1_main:\n - rtsp://admin:PASSWORD@192.168.1.30:554/cam/realmonitor?channel=1&subtype=0\n tree_camera_1_sub:\n - rtsp://admin:PASSWORD@192.168.1.30:554/cam/realmonitor?channel=1&subtype=1\n garage_camera_main:\n - rtsp://admin:PASSWORD@192.168.1.31:554/cam/realmonitor?channel=1&subtype=0\n garage_camera_sub:\n - rtsp://admin:PASSWORD@192.168.1.31:554/cam/realmonitor?channel=1&subtype=1\n\ncameras:\n garage_camera:\n enabled: true\n ffmpeg:\n inputs:\n - path: rtsp://127.0.0.1:8554/garage_camera_main\n roles:\n - record\n - path: rtsp://127.0.0.1:8554/garage_camera_sub\n roles:\n - detect\n detect:\n enabled: true\n\n tree_camera_1:\n enabled: true\n ffmpeg:\n inputs:\n - path: rtsp://127.0.0.1:8554/tree_camera_1_main\n roles:\n - record\n - path: rtsp://127.0.0.1:8554/tree_camera_1_sub\n roles:\n - detect\n detect:\n enabled: true\n\nversion: 0.17-0\n/mnt/Storage/frigaterootwheel# Start Frigate\ndocker compose up -d\n\n# Verify NFS mount inside container (should show NAS storage)\ndocker exec frigate df -h /media/frigate\n# :/mnt/Storage/frigate 22T 0 22T 0% /media/frigate\n\n# Check logs\ndocker logs frigate --tail 20\n\n# Check storage usage\ndocker exec frigate du -sh /media/frigate/recordings/\nFrigate web UI available at http://localhost:8971.
\nThe system is running. Two cameras are up, recording continuously to the NAS, detecting people and cars with AI. It works. But there's more to do:
\nWas this overkill for replacing Blink cameras? Absolutely. But I got to brush up on fishing cable through an attic and cementing PVC conduit, while picking up NFS volume configuration in Docker, RTSP authentication debugging across three layers of URL encoding, and setting up an AI surveillance system that doesn't phone home to anyone.
\nThe Blink cameras are in a drawer. Good riddance.
\nAfter all the YAML wrestling and permission-denied debugging, it's nice to have something physical to point at. Here's the real install.
First camera went up on a tree in the front yard, high enough that nobody can mess with it but low enough to actually see things. Cable runs down the trunk inside conduit so it doesn't look like a science fair project from the street.
The garage camera tucks into the soffit corner, watching the side yard and driveway. White housing on white trim, you forget it's there until a detection alert lights up your phone.
![\"\"](\"https://emir.fyi/content/images/2026/04/garage-camera-1.webp\")
The least softwary part of the whole project. PVC conduit, a tube of adhesive, some fishing tape, and a willingness to climb a ladder. Honestly the most satisfying few hours of the build.
![\"\"](\"https://emir.fyi/content/images/2026/04/conduit-soffit-run-1.webp\")
First time using a PoE switch and I'm a convert. One cable to each camera, no wall warts, no power injectors strapped to the wall. The TP-Link slots into the rack with the rest of the gear and quietly does its job.
![\"\"](\"https://emir.fyi/content/images/2026/04/poe-switch-rack.webp\")
![\"\"](\"https://emir.fyi/content/images/2026/04/m4-on-m1-base.webp\")
The Frigate workhorse is the bottom Mac Mini in this photo. The silver one stacked on top is an M4 that joined the rack later for unrelated reasons. The M1 didn't get demoted when the M4 showed up, it got promoted to \"permanent appliance.\" It just sits there, sips watts, runs Frigate.
","comment_id":"69b2c54ae62aae000136c0a3","feature_image":"https://emir.fyi/content/images/2026/03/ChatGPT-Image-Mar-12--2026--10_50_51-AM.png","featured":false,"visibility":"public","created_at":"2026-03-12T09:53:14.000-04:00","updated_at":"2026-04-15T20:00:20.000-04:00","published_at":"2026-03-12T10:44:23.000-04:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/running-frigate-nvr-on-m1-mac-mini-with-truenas-nfs-storage/","excerpt":"Why did I do this\n\n\nI got tired of changing Blink camera batteries and of the thought that someone in the cloud might have recordings of my family's every move.\n\n\nThat's it. That's the origin story. Every now and then I'd get the notification, trudge outside, swap batteries, and wonder why I was paying a cloud subscription for the privilege of watching grainy clips of my own front yard on someone else's server. So I decided to fix it. Overkill style.\n\n\nI'm a homelabber. I build things to learn t","reading_time":14,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"69b57464cbda360001756c8b","uuid":"0a71c4a1-c29c-4e40-bc68-17c1890c7dd4","title":"Entire Homelab Dies Because One Machine Reboots: Building an HA Load Balancer with Keepalived and Caddy","slug":"entire-homelab-dies-because-one-machine-reboots-building-an-ha-load-balancer-with-keepalived-and-caddy","html":"My Proxmox host rebooted, and everything went dark.
\nVaultwarden, ArgoCD, my monitoring dashboard, the blog, Paperless. All of it. Every service in my homelab routed through a Kubernetes cluster running on Proxmox VMs, and the load balancer for that cluster was MetalLB, which also ran inside Kubernetes. So when Proxmox went down for a firmware update, the load balancer died with it, which meant the ingress died, which meant DNS entries pointing at the ingress went nowhere. A planned five-minute reboot turned into a \"why is everything broken\" moment for anyone in the house trying to use anything I host.
\nThe irony was thick. I had a K8s cluster with multiple worker nodes, redundant pod replicas, health checks, the works. But none of that matters when your load balancer is a pod inside the same cluster it's supposed to be balancing. It's like putting the pilot's ejection seat inside the engine.
\nMetalLB is a bare-metal Kubernetes load balancer. It gives your services a real IP address on your LAN using ARP announcements, so you can hit 192.168.1.201 and reach your Traefik ingress controller. For a lot of homelabs, this is perfectly fine.
Here's the problem: MetalLB runs as pods inside K8s. If K8s goes down, MetalLB goes down, and that IP stops responding. There's no failover outside the cluster. Your DNS entries still point at 192.168.1.201, but nobody's home.
I also had services running outside K8s entirely. SiYuan (notes), Paperless (document management), and my Ghost blog all run as Docker containers on a separate Portainer host. They were routed through K8s using an external-services pattern (headless Service + Endpoints pointing at the Docker host IP). This worked, but it meant that even non-K8s services depended on K8s being alive. A notes app shouldn't go offline because Kubernetes is having a bad day.
\nAnd then there's Beszel (my monitoring dashboard) and Frigate (security cameras), both running natively on the Mac Mini M1 that has nothing to do with Proxmox or K8s. These could survive a Proxmox outage entirely, but I couldn't reach them because the routing went through the dead cluster.
\nVRRP (Virtual Router Redundancy Protocol) is the simplest answer to \"what if the machine holding the IP goes down?\" Multiple nodes run keepalived, each with a priority. The highest priority healthy node becomes the master and owns the virtual IP (VIP). If the master dies, the next highest takes over. Usually within a second.
\nNo coordination service. No consensus algorithm. No etcd. Just VRRP advertisements over multicast, every second. The nodes don't even need to know about each other's existence. They just broadcast \"I'm alive and my priority is X\" and the protocol handles the rest.
\nI needed a reverse proxy on each LB node that could:
\nCaddy does all of this out of the box. The Caddyfile is readable by humans, TLS just works if you hand it a cert and key, and it has built-in health checking for upstreams. I already use Traefik inside K8s, but Caddy felt like the right tool for a standalone reverse proxy that should be as boring and reliable as possible.
\nI looked at HAProxy and Nginx too. HAProxy would work but the config syntax is verbose for what I need. Nginx would also work but I'd have to manually configure health checks and TLS in a way that Caddy just handles automatically. For three nodes running identical configs, simplicity wins.
\nTwo nodes give you failover. Three give you failover without anxiety. If one of the Mac Minis is off for maintenance and then Proxmox reboots, you still have a working load balancer. The cost is minimal: each LB node runs Ubuntu with 512MB RAM and 1 CPU core. Three lightweight VMs spread across three physical machines.
\n| Node | \nIP | \nPhysical Host | \nPriority | \nRole | \n
|---|---|---|---|---|
| lb-m4 | \n192.168.1.22 | \nMac Mini M4 Pro (UTM VM) | \n100 | \nDefault master | \n
| lb-proxmox | \n192.168.1.20 | \nProxmox (QEMU VM) | \n90 | \nBackup | \n
| lb-m1 | \n192.168.1.23 | \nMac Mini M1 (UTM VM) | \n80 | \nBackup | \n
VIP: 192.168.1.221/24 (outside the DHCP range)
The Mac Mini M4 Pro gets priority 100 because it's the beefy daily driver that's always on. The Proxmox VM gets 90 because it's the most likely to go down during maintenance. The Mac Mini M1 gets 80 as the last resort.
\nHere's what the traffic flow looks like now:
\n LAN devices\n (laptops, phones,\n other machines)\n |\nInternet --> Cloudflare |\n --> cloudflared |\n | |\n | *.localdomain |\n | DNS resolves |\n | to VIP |\n v v\n HA Load Balancer (VIP 192.168.1.221)\n keepalived + Caddy, 3 nodes\n |\n +-- K8s services (vault, argocd, market-mind)\n | Caddy --> Traefik NodePort 30443\n | Round-robin across K8s workers\n |\n +-- Portainer services (siyuan, paperless, errbit)\n | Caddy --> direct to 192.168.1.2\n |\n +-- Mac Mini M1 services (metrics, frigate)\n | Caddy --> direct to 192.168.1.5\n | Fully HA: survives Proxmox outage\n |\n +-- Ghost blog (:80)\n cloudflared --> Caddy --> 192.168.1.2:2368\nMost traffic isn't coming from the internet. It's coming from inside the house. Laptops opening Vaultwarden to grab a password. Phones checking Frigate camera feeds. My dev machine hitting ArgoCD or the monitoring dashboard. Other services calling each other over *.localdomain hostnames. The router resolves all *.localdomain DNS entries to the VIP (192.168.1.221), so every device on the network hits Caddy the same way an external request would, just without the Cloudflare hop. Public traffic (the blog, MarketMind) comes in through Cloudflare and cloudflared, but that's a small fraction of the total load.
The key insight: Caddy doesn't just proxy to K8s. It routes everything by hostname. K8s services go through Traefik's NodePort, Docker services go directly to the Portainer host, and Mac Mini M1 services go directly to that machine. Each class of service has a different failure domain. If Proxmox goes down, you lose K8s and Portainer services, but metrics and Frigate stay up because they live on the Mac Mini M1, a completely separate physical machine.
\nThis was the straightforward node. Clone an Ubuntu template, assign a static IP via cloud-init, SSH in, done. I use Terraform to provision VMs on Proxmox, so lb-proxmox is defined as infrastructure-as-code alongside the K8s nodes:
\nresource \"proxmox_vm_qemu\" \"lb_node\" {\n name = \"lb-node\"\n target_node = \"proxmox\"\n vmid = 105\n clone = \"ubuntu-template\"\n full_clone = true\n agent = 1\n onboot = true\n cores = 1\n memory = 512\n\n disk {\n type = \"disk\"\n slot = \"scsi0\"\n size = \"8G\"\n storage = \"local-lvm\"\n }\n\n network {\n id = 0\n model = \"virtio\"\n bridge = \"vmbr0\"\n }\n\n ipconfig0 = \"ip=192.168.1.20/24,gw=192.168.1.1\"\n nameserver = \"192.168.1.1\"\n ciuser = \"ubuntu\"\n sshkeys = file(\"~/.ssh/my_key.pub\")\n}\nUTM is a QEMU-based hypervisor for Apple Silicon. It runs ARM64 Linux VMs natively, which is exactly what I needed for the Mac Mini M4 Pro and Mac Mini M1 nodes.
\nThe catch: UTM doesn't have cloud-init, Terraform, or any automation story. You install Ubuntu from an ISO, manually configure networking, and SSH in (scp your public key into authorized_keys). For the Mac Mini M1, which runs headless in a closet, I had to do the whole install over screen sharing, then set up a LaunchAgent to auto-start the VM on boot.
\nBridged networking is critical. The VM needs to be on the same L2 network as the other LB nodes for VRRP to work. In UTM, this means selecting the wired Ethernet adapter (not WiFi) as the bridged interface. On Mac, en0 is wired and en1 is WiFi. Getting this backwards means your VM gets an IP but VRRP advertisements never reach the other nodes.
Auto-start on boot uses a LaunchAgent plist that waits 15 seconds (for UTM to finish launching) then runs utmctl start:
<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\"\n \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n<plist version=\"1.0\">\n<dict>\n <key>Label</key>\n <string>com.utm.autostart-m4-lb</string>\n <key>ProgramArguments</key>\n <array>\n <string>/bin/bash</string>\n <string>-c</string>\n <string>sleep 15 && /Applications/UTM.app/Contents/MacOS/utmctl start m4-lb</string>\n </array>\n <key>RunAtLoad</key>\n <true/>\n</dict>\n</plist>\nDrop this in ~/Library/LaunchAgents/, enable auto-login on the Mac, and disable FileVault. The VM comes up about 30 seconds after boot. Not instant, but keepalived handles the gap. The other nodes own the VIP until this one joins.
Each node runs the same keepalived config, templated by Ansible with per-node priority and interface:
\nvrrp_instance VI_1 {\n state BACKUP\n interface {{ lb_interface }}\n virtual_router_id 51\n priority {{ lb_priority }}\n advert_int 1\n\n authentication {\n auth_type PASS\n auth_pass {{ lb_vrrp_pass }}\n }\n\n virtual_ipaddress {\n 192.168.1.221/24 dev {{ lb_interface }}\n }\n}\nEvery node starts as BACKUP. The one with the highest priority wins the election and becomes master. advert_int 1 means nodes advertise every second. If the master misses three advertisements (3 seconds), failover happens. In practice, I see sub-second failover in my testing.
All nodes use the same virtual_router_id (51) and authentication password so they recognize each other as part of the same VRRP group. The interface varies: eth0 on the Proxmox VM, enp0s1 on the UTM VMs.
This is the fun part. The Caddyfile is identical on all three nodes, templated once and deployed everywhere:
\n# Reusable snippet for K8s services via Traefik NodePort\n(k8s_backend) {\n reverse_proxy 192.168.1.11:30443 192.168.1.12:30443 {\n transport http {\n tls\n tls_insecure_skip_verify\n }\n lb_policy round_robin\n health_interval 10s\n health_timeout 5s\n }\n}\n\n# K8s services\nvault.localdomain {\n tls /etc/caddy/certs/wildcard.crt /etc/caddy/certs/wildcard.key\n import k8s_backend\n}\n\nargocd.localdomain {\n tls /etc/caddy/certs/wildcard.crt /etc/caddy/certs/wildcard.key\n import k8s_backend\n}\n\nmarket-mind.localdomain {\n tls /etc/caddy/certs/wildcard.crt /etc/caddy/certs/wildcard.key\n import k8s_backend\n}\n\n# Docker/Portainer services (direct, no K8s involved)\nsiyuan.localdomain {\n tls /etc/caddy/certs/wildcard.crt /etc/caddy/certs/wildcard.key\n reverse_proxy 192.168.1.2:6806\n}\n\npaperless.localdomain {\n tls /etc/caddy/certs/wildcard.crt /etc/caddy/certs/wildcard.key\n reverse_proxy 192.168.1.2:28981\n}\n\n# Fully HA services (Mac Mini M1, survives Proxmox outage)\nmetrics.localdomain {\n tls /etc/caddy/certs/wildcard.crt /etc/caddy/certs/wildcard.key\n reverse_proxy 192.168.1.5:8090\n}\n\n# Public HTTP (Ghost blog via cloudflared)\n:80 {\n reverse_proxy 192.168.1.2:2368 {\n header_up X-Forwarded-Proto https\n }\n}\nA few things worth noting:
\nThe k8s_backend snippet is a Caddy named snippet. Every K8s service imports it, so adding a new one is a single block with import k8s_backend. The snippet round-robins across both K8s worker nodes on port 30443 (Traefik's NodePort) with health checks every 10 seconds. If a worker goes down, Caddy stops routing to it automatically.
tls_insecure_skip_verify is needed because Traefik presents a self-signed certificate on the NodePort. Caddy is talking to Traefik over the LAN, so verifying Traefik's cert isn't necessary. The wildcard cert from step-ca is what clients (browsers) see.
The X-Forwarded-Proto: https header on port 80 prevents Ghost from going into an infinite redirect loop. Ghost sees HTTP from Caddy, tries to redirect to HTTPS, Cloudflare sends it back as HTTP, Ghost redirects again. Setting the header tells Ghost \"the client is already on HTTPS, stop redirecting.\"
metrics.localdomain points directly at the Mac Mini M1. This service has zero dependency on Proxmox, K8s, or Docker. If Proxmox burns to the ground, my monitoring dashboard stays up. That felt important.
With Caddy handling ingress, Traefik no longer needs a MetalLB LoadBalancer IP. Instead, it exposes a NodePort service that Caddy routes to:
\napiVersion: v1\nkind: Service\nmetadata:\n name: traefik-nodeport\n namespace: traefik\nspec:\n type: NodePort\n selector:\n app.kubernetes.io/name: traefik\n ports:\n - name: web\n port: 80\n targetPort: web\n nodePort: 30080\n - name: websecure\n port: 443\n targetPort: websecure\n nodePort: 30443\nTraefik still does all the K8s-internal routing (IngressRoutes, middleware, TLS termination for K8s services). It just doesn't own a LAN IP anymore. Caddy owns the LAN IP (via the VIP) and forwards to Traefik on the NodePort.
\nI left the old LoadBalancer service in place. It shows <pending> now that MetalLB is gone, but it's harmless and removing it would require updating the Helm values.
Three nodes running identical configs is the perfect Ansible use case. The entire deployment is a single playbook:
\n- name: Deploy HA load balancer (keepalived + Caddy)\n hosts: lb_nodes\n become: true\n tasks:\n - name: Install keepalived and caddy\n ansible.builtin.apt:\n name: [keepalived, caddy]\n state: present\n update_cache: true\n\n - name: Create cert directory\n ansible.builtin.file:\n path: /etc/caddy/certs\n state: directory\n owner: caddy\n group: caddy\n mode: \"0750\"\n\n - name: Copy wildcard TLS certificate\n ansible.builtin.copy:\n src: \"{{ lb_cert_src }}\"\n dest: /etc/caddy/certs/wildcard.crt\n owner: caddy\n group: caddy\n mode: \"0640\"\n notify: Reload caddy\n\n - name: Copy wildcard TLS key\n ansible.builtin.copy:\n src: \"{{ lb_key_src }}\"\n dest: /etc/caddy/certs/wildcard.key\n owner: caddy\n group: caddy\n mode: \"0640\"\n notify: Reload caddy\n\n - name: Template Caddyfile\n ansible.builtin.template:\n src: templates/Caddyfile.j2\n dest: /etc/caddy/Caddyfile\n notify: Reload caddy\n\n - name: Template keepalived config\n ansible.builtin.template:\n src: templates/keepalived.conf.j2\n dest: /etc/keepalived/keepalived.conf\n notify: Restart keepalived\n\n - name: Enable and start services\n ansible.builtin.systemd:\n name: \"{{ item }}\"\n enabled: true\n state: started\n loop: [caddy, keepalived]\n\n handlers:\n - name: Reload caddy\n ansible.builtin.systemd:\n name: caddy\n state: reloaded\n\n - name: Restart keepalived\n ansible.builtin.systemd:\n name: keepalived\n state: restarted\nThe inventory defines per-node variables:
\nlb_nodes:\n hosts:\n 192.168.1.20:\n ansible_user: ubuntu\n lb_priority: 90 # Proxmox VM\n lb_interface: eth0\n 192.168.1.23:\n ansible_user: ubuntu\n lb_priority: 80 # Mac Mini M1 UTM VM\n lb_interface: enp0s1\n 192.168.1.22:\n ansible_user: ubuntu\n lb_priority: 100 # Mac Mini M4 Pro UTM VM\n lb_interface: enp0s1\nOne command deploys or updates all three nodes:
\nansible-playbook -i inventory.yml playbook-ha-lb.yml --extra-vars '@lb-keys.yml'\nThe lb-keys.yml file (gitignored) contains the VRRP shared secret and paths to the TLS cert/key from step-ca. The cert gets copied from my Mac (which acts as an SCP relay between step-ca and the LB nodes).
The actual cutover from MetalLB to the HA LB was surprisingly smooth. Here's what happened:
\n*.localdomain hostnames at 192.168.1.221 (the VIP) instead of 192.168.1.201 (old MetalLB IP)The only downtime was the DNS change, which propagates instantly on my router since it's the authoritative DNS for .localdomain. Total switchover: about 30 seconds.
This is the part I actually tested. Repeatedly. By pulling network cables and watching what happened.
\nProxmox reboots (the original problem):
\nK8s services go down (expected, VMs are on Proxmox), lb-proxmox goes down. But lb-m4 or lb-m1 keeps the VIP alive. When Proxmox comes back, lb-proxmox rejoins automatically and starts receiving traffic again. Metrics and Frigate never went down because they live on the Mac Mini M1.
Mac Mini M4 Pro goes offline (highest priority node):
\nlb-proxmox (priority 90) takes over the VIP within ~1 second. All services continue working. When lb-m4 comes back, it reclaims the VIP (preemption is enabled by default in keepalived).
Two nodes down simultaneously:
\nThe remaining node owns the VIP alone. Services routed to the surviving backends still work. This is why three nodes across three physical machines matters. The odds of all three dying at once are vanishingly small.
Complete power outage:
\nEverything dies. But when power returns, the Mac Minis auto-login, UTM launches, the LaunchAgents start the VMs, keepalived elects a master, and the VIP comes up. No manual intervention. I tested this by flipping the breaker. It took about 90 seconds from power-on to VIP-active.
The biggest lesson (That I already knew and applied only at the job): your load balancer can't live inside the thing it's balancing. This seems obvious in retrospect, but MetalLB makes it so easy to set up that you don't think about it until it bites you. For production Kubernetes clusters backed by cloud providers, this isn't an issue because the cloud LB is external. For homelabs on bare metal, you have to build that external layer yourself.
\nKeepalived is ridiculously simple for what it does. The entire config is 16 lines. VRRP is a solved protocol from the 1990s and it just works. No service discovery, no leader election complexity, no distributed state. Three daemons yelling their priority over the network every second. The loudest healthy one wins. Beautiful.
\nCaddy replaced an entire Kubernetes routing layer (MetalLB + external-services pattern) with a single config file. I went from \"headless Service with manually maintained Endpoints pointing at a LAN IP, routed through an IngressRoute on Traefik that I need MetalLB to expose\" to \"reverse_proxy 192.168.1.2:6806\". Seven lines of YAML became one line of Caddyfile.
\nSpreading nodes across physically separate machines is the real HA move. If all three VMs were on Proxmox, this entire exercise would have been pointless. The Mac Minis are the insurance policy. They're low-power, always-on machines that don't share a failure domain with the hypervisor. One of them even hosts services that are completely Proxmox-independent. That's the setup I should have had from day one.
\nAnd UTM on Apple Silicon is surprisingly capable for running lightweight Linux VMs. It's not Proxmox, it's not VMware, and it has zero automation story. But for a 512MB Ubuntu VM running keepalived and Caddy, it's perfectly adequate. Just budget extra time for the initial setup if the Mac is headless.
\n","comment_id":"69b57464cbda360001756c8b","feature_image":"https://emir.fyi/content/images/2026/03/ChatGPT-Image-Mar-14--2026--11_05_33-AM.png","featured":false,"visibility":"public","created_at":"2026-03-14T10:44:52.000-04:00","updated_at":"2026-03-14T11:05:47.000-04:00","published_at":"2026-02-18T11:02:00.000-05:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/entire-homelab-dies-because-one-machine-reboots-building-an-ha-load-balancer-with-keepalived-and-caddy/","excerpt":"Why I Did This\n\n\nMy Proxmox host rebooted, and everything went dark.\n\n\nVaultwarden, ArgoCD, my monitoring dashboard, the blog, Paperless. All of it. Every service in my homelab routed through a Kubernetes cluster running on Proxmox VMs, and the load balancer for that cluster was MetalLB, which also ran inside Kubernetes. So when Proxmox went down for a firmware update, the load balancer died with it, which meant the ingress died, which meant DNS entries pointing at the ingress went nowhere. A pl","reading_time":12,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"697d77cfb97a8100019a9bda","uuid":"7f34baab-4a23-48a0-a743-13e78f7429f2","title":"Upgrading PostgreSQL from version 14 to 17 - Part 2","slug":"upgrading-postgresql-from-version-14-to-17-part-1-2","html":"Coming soon... maybe
","comment_id":"697d32c550c39300018cc931","feature_image":"https://emir.fyi/content/images/2026/01/ChatGPT-Image-Jan-30--2026--02_05_40-PM-1.png","featured":false,"visibility":"public","created_at":"2026-01-30T17:37:57.000-05:00","updated_at":"2026-01-30T17:38:59.000-05:00","published_at":"2026-01-30T17:38:21.000-05:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/upgrading-postgresql-from-version-14-to-17-part-1-2/","excerpt":"Coming soon... maybe","reading_time":0,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null},{"id":"697d77cfb97a8100019a9bd9","uuid":"1e93e8bc-64c6-46f8-9a6d-851a8c42cbbc","title":"Upgrading PostgreSQL from version 14 to 17 - Part 1","slug":"upgrading-postgresql-from-version-14-to-17-part-1","html":"To start I should probably mention that I work for a healthcare company, and our PostgreSQL databases are hosted on Aptible.
Also this is NOT a tutorial. This is me writing about an experience I went through, which involved a fair amount of technical work, along with the usual emotional and everyday realities that come with it. Now that I have that out of the way, let's begin.
Our database is very write heavy. Incoming data is arriving from batch import jobs that run on demand and over-night. The outgoing data is typically consumed via the web app (some via API too), concentrated during normal PST work hours, with very little activity outside business hours or on weekends.
Database hardware:
There are no failovers or read replicas configured. This is a single, chunky database.
Now for the reason we upgraded. It wasn't performance, new features, or anything particularly exciting (though we'll happily take the benefits of a newer version). The real reason was reducing cost. Once you factor in database backups required for disaster recovery and compliance, that unused space gets expensive very quickly.
Upgrading PostgreSQL on Aptible is usually straightforward. They offer several upgrade paths, and the easiest and best is an in-place upgrade. Unfortunately, that option wasn't available to us. Some underlying library binaries changed between these versions, which ruled it out. Major bummer.
That left two options: logical replication or dump and restore. At first glance, logical replication was the better choice. Downtime would be minimal compared to dump and restore. With the latter, we would need to stop the database entirely, or at least stop writes.
Next, we needed to do a dry run of the migration because when have things ever worked on the first try, especially at this scale. Data consistency and reliability are very important to us, arguably more important than availability. That said, like any business, we have customers and contracts to sign SLAs to meet, and all the usual constraints that come with that.
Going back to the actual work, I started by putting together a document to capture the steps and a few observations along the way. I won't go into too much detail here, but it looked roughly like this:
I was genuinely excited to work on this. It was the end of the year, and if everything went well, we'd start the new year paying nearly 50% less for the database while also running a newer and faster version of PostgreSQL.
With the plan written down and reviewed by a peer (minor feedback, green light), I moved on to actually running the thing. Fast forward to kicking off aptible db:replicate to create the replica and start replication and… khaboom.
This is the error I immediately ran into:
INFO -- : ║ 2025-11-25 18:16:25 UTC HINT: You might need to increase \"max_locks_per_transaction\".\nINFO -- : ║ pg_restore: error: could not execute query: ERROR: out of shared memory\nINFO -- : ║ HINT: You might need to increase \"max_locks_per_transaction\".\nINFO -- : ║ Command was: CREATE SEQUENCE schema1.activities_id_seq\nINFO -- : ║ START WITH 1\nINFO -- : ║ INCREMENT BY 1\nINFO -- : ║ NO MINVALUE\nINFO -- : ║ NO MAXVALUE\nINFO -- : ║ CACHE 1;Given how many schemas and database objects we have, this wasn't entirely surprising. With the default settings, the replica simply couldn't be provisioned within a single transaction. We were blowing past the default limits.
To be clear about what I mean by object count, this is what I was looking at:
SELECT count(*) \nFROM pg_class \nWHERE relkind IN ('r','p','i','S','t','m','v','I');The obvious fix here is to increase max_locks_per_transaction (and max_pred_locks_per_transaction), so that's what I did. I bumped them up significantly from the default value of 64. I even took a screenshot so I'd remember exactly how far I pushed them.
![\"\"](\"https://emir.fyi/content/images/2026/01/db-settings.png\")
And yet the replica still failed to provision. At that point, I reached out to Aptible support.
They were friendly and responsive, but the back-and-forth dragged on. \"Try increasing it more.\" Then more. Try this, try that. Then bump up even more. This went on for about a week. I'm compressing the exchange here, but troubleshooting something like this over email is hard enough, doing it live in production is even harder. Not trying to talk down on Aptible support here and henceforth.
Eventually, we jumped on a call (I should have done that sooner). That's when things finally clicked.
I was able to demonstrate that configuration changes I was applying to the source database were not being applied to the replica database at all. Every attempt, every tweak, every retry had been happening against a replica that never picked up the new settings.
Once they realized that, everything suddenly made sense. And honestly, it was great news.
![\"\"](\"https://emir.fyi/content/images/2026/01/Screenshot-2026-01-30-at-4.30.27---PM.png\")
After a couple more status updates over email, and one Thanksgiving holiday later, I got the notice that the new release had shipped:
![\"\"](\"https://emir.fyi/content/images/2026/01/Screenshot-2026-01-30-at-4.59.55---PM.png\")
That was my cue to start the replication test immediately. I followed the steps from the document I had put together earlier, and for the first day or so everything looked fine. Initialization was progressing as expected. I was keeping an eye on it using a small script for tracking progress, in case you're curious. Then, the very next day an unwelcome surprise.
![\"\"](\"https://emir.fyi/content/images/2026/01/Screenshot-2026-01-30-at-5.11.19---PM.png\")
Reading that update was frustrating. The issue only affected newly created replicas, not the one I was actively testing, but it still clearly needed to be fixed. This wasn't something we could just ignore. A day or two later, while replication was still ongoing, another email landed.:
![\"\"](\"https://emir.fyi/content/images/2026/01/Screenshot-2026-01-30-at-5.14.00---PM.png\")
At that point, I wasn't confident this was going to get resolved quickly enough for us to complete the upgrade over the holidays. Problems like this are hard, and as a customer you only ever see your own use case. There are always hidden complexities, especially the ones you don't have visibility into. Because of that, I empathize with the Aptible support and engineering teams.
I let my manager know that I was starting work on an alternative plan for the upgrade. This was supposed to be a side quest. It was clearly turning into the main one.
That's it for part one. If this was interesting and you're curious about what happened next, part two will cover that (maybe).
I used ChatGPT to help clean up grammar, since I'm not a native speaker. The ideas, opinions, and descriptions are mine. The header image was generated using this prompt:
I want you to generate image with postgresql logo in it and somehow we paint over 14 number 17, background needs to be dark because my web styling is white.
","comment_id":"697cfd5b50c39300018cc822","feature_image":"https://emir.fyi/content/images/2026/01/ChatGPT-Image-Jan-30--2026--02_05_40-PM.png","featured":false,"visibility":"public","created_at":"2026-01-30T13:50:03.000-05:00","updated_at":"2026-04-15T19:23:46.000-04:00","published_at":"2026-01-30T17:28:38.000-05:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/upgrading-postgresql-from-version-14-to-17-part-1/","excerpt":"To start I should probably mention that I work for a healthcare company, and our PostgreSQL databases are hosted on Aptible.\n\nAlso this is NOT a tutorial. This is me writing about an experience I went through, which involved a fair amount of technical work, along with the usual emotional and everyday realities that come with it. Now that I have that out of the way, let's begin.\n\nOur database is very write heavy. Incoming data is arriving from batch import jobs that run on demand and over-night. ","reading_time":6,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":"It was a piece of cake in my mind"},{"id":"69e239d43cad350001301c89","uuid":"69e4252c-c10e-44a1-9af8-d08065787e95","title":"Turning Paperless-ngx Into a Smart Filing Cabinet With Local AI and NAS Storage","slug":"turning-paperless-ngx-into-a-smart-filing-cabinet-with-local-ai-and-nas-storage","html":"I had Paperless-ngx running. Technically. It was deployed on Portainer, had a DNS entry, a TLS cert, the whole nine yards. It also had zero documents in it. It was a ghost town with a login page. Every time I'd get a medical bill, a receipt, or a tax document, I'd do what every responsible adult does: take a photo on my phone and forget which album it ended up in.
\nThe straw that broke it was tax season. Every year, my wife and I spend hours across multiple days going through all of her business bank account statements, manually categorizing every expense. Is this one deductible? Was that a business lunch or a personal one? What about that $47 charge from November that neither of us remembers? It's tedious, error prone, and we both dread it. The whole time I'm thinking: I have a rack of computers in the other room. There has to be a better way. That's when I decided to actually use Paperless. Not just run it. Use it. The goal is simple: upload receipts and invoices throughout the year, let AI tag them as tax-deductible or not, and when tax season comes around, just filter by the tag instead of excavating bank statements.
\nBut basic Paperless with its built-in OCR was not going to cut it. I'd seen what happens when you feed it a camera photo of a receipt. Tesseract (the OCR engine) does its best, but \"its best\" for a photo of crumpled paper under fluorescent lighting is a string of characters that looks like someone fell asleep on a keyboard. I wanted the documents to be searchable, auto categorized, and ideally smart enough to tag a medical bill as \"medical\" and a hosting invoice as \"tax-deductible\" without me lifting a finger.
\nI watched Techno Tim's video where he set up Paperless-ngx with Ollama and a couple of AI add-ons that dramatically improved both the metadata and the OCR quality. Great video. But his setup runs everything in one big Docker Compose stack on a single Linux box with an NVIDIA GPU. My homelab is a bit more distributed, so the architecture had to adapt.
\n| Techno Tim's Setup | \nMy Setup | \n
|---|---|
| Single Linux server with NVIDIA GPU | \nLXC container (Portainer) + Mac Mini M4 Max | \n
| Everything in one Docker Compose | \nServices split across hosts | \n
| Ollama in Docker (NVIDIA GPU passthrough) | \nOllama native on macOS (Metal GPU, no Docker) | \n
| Local storage | \nTrueNAS NAS via NFS | \n
The big deviation: Docker on Apple Silicon cannot pass through the GPU to containers. So Ollama has to run natively on macOS to use the M4 Max's Metal acceleration. Everything else stays containerized.
\nThink of it like this: Paperless-ngx is the filing cabinet, Ollama is the brain running on a separate machine, and two AI add-ons (paperless-ai and paperless-gpt) sit in between making everything smarter.
\nPaperless-ngx stores, indexes, and searches documents. It runs built-in OCR via Tesseract and handles the web UI where you interact with everything.
\nGotenberg converts non-PDF documents (Word, Excel, HTML, emails) into PDFs so Paperless can process them. Without it, you're limited to PDFs and images.
\nTika (Apache Tika) extracts metadata and text from various file formats. It reads the guts of Office docs, emails, and other formats so Paperless can index them.
\nOllama runs the actual LLM models. It sits on the M4 Max, listening on port 11434, serving two models: qwen3:8b for text processing and minicpm-v:8b for vision (understanding images).
paperless-ai reads the text content that Paperless already extracted and feeds it to the LLM. The LLM assigns tags, generates titles, detects correspondents (who sent it), and classifies the document type. It's the smart filing clerk.
\npaperless-gpt replaces the OCR itself. Instead of Tesseract guessing at pixels, it feeds the document image to a vision model (minicpm-v) that actually understands what it's looking at. Tables become tables, serial numbers stay intact, and blurry text gets read correctly.
\nHere's how it all connects:
\nDocument uploaded\n → Paperless-ngx ingests + basic OCR (Tesseract)\n → Gotenberg converts non-PDFs\n → Tika extracts metadata\n → paperless-ai reads text → sends to Ollama (qwen3:8b) → writes back tags/title/type\n → paperless-gpt reads image → sends to Ollama (minicpm-v:8b) → replaces OCR content\nThis tripped me up. Techno Tim runs Ollama in Docker with --gpus all because he has an NVIDIA GPU where Docker GPU passthrough just works. On Apple Silicon, Docker Desktop runs a Linux VM under the hood, and that VM has no access to the Metal GPU. Running Ollama in Docker on a Mac means CPU-only inference, which is painfully slow for any useful model.
The solution: install Ollama natively on macOS. It automatically uses Metal for GPU acceleration, and the M4 Max's unified memory architecture means models load fast and run fast. I got 48 tokens/second on minicpm-v:8b, which is plenty for document processing.
\nI wrote an Ansible playbook to automate the install, configure it to listen on all interfaces (so containers on other hosts can reach it), and pull the models:
\n# ansible/playbook-ollama.yml (abbreviated)\n- name: Install and configure Ollama on macOS (Apple Silicon)\n hosts: mac_ai\n tasks:\n - name: Install Ollama via Homebrew\n community.general.homebrew:\n name: ollama\n state: present\n\n - name: Create Ollama LaunchAgent plist\n ansible.builtin.template:\n src: templates/com.ollama.serve.plist.j2\n dest: \"{{ ansible_facts.env.HOME }}/Library/LaunchAgents/com.ollama.serve.plist\"\n\n - name: Pull qwen3:8b model\n ansible.builtin.command: /opt/homebrew/bin/ollama pull qwen3:8b\n\n - name: Pull minicpm-v:8b vision model\n ansible.builtin.command: /opt/homebrew/bin/ollama pull minicpm-v:8b\nThe LaunchAgent plist configures OLLAMA_HOST=0.0.0.0:11434 so other machines on the LAN can reach it, and KeepAlive=true so it auto-restarts if it crashes:
<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<plist version=\"1.0\">\n<dict>\n <key>Label</key>\n <string>com.ollama.serve</string>\n <key>ProgramArguments</key>\n <array>\n <string>/opt/homebrew/bin/ollama</string>\n <string>serve</string>\n </array>\n <key>EnvironmentVariables</key>\n <dict>\n <key>OLLAMA_HOST</key>\n <string>0.0.0.0:11434</string>\n <key>OLLAMA_KEEP_ALIVE</key>\n <string>15m</string>\n </dict>\n <key>RunAtLoad</key>\n <true/>\n <key>KeepAlive</key>\n <true/>\n</dict>\n</plist>\nI wanted documents stored on my TrueNAS NAS, not on the Portainer host's local disk. NAS gives me ZFS snapshots, redundancy, and one place to back up. I'd already worked with NFS in the past, so I figured this would be quick. It was not quick.
\nThe Paperless container runs on Portainer, which is an unprivileged LXC container on Proxmox. And unprivileged LXC containers cannot mount filesystems. The mount() syscall is blocked at the kernel level, even as root inside the container. I spent a while tweaking NFS export settings on TrueNAS before realizing the request was never leaving the container.
The standard pattern for LXC + NFS:
\nmujo-nas.localdomain is DNS for my NAS.On Proxmox:
\n# /etc/fstab on Proxmox host\nmujo-nas.localdomain:/mnt/Storage/paperless /mnt/nas/paperless nfs vers=3,nolock,soft,rw,_netdev 0 0\nIn the LXC config (/etc/pve/lxc/100.conf):
mp0: /mnt/nas/paperless,mp=/mnt/nas/paperless\nOne catch with unprivileged LXCs: UID namespacing. Container UID 1000 maps to host UID 101000, so files created by the container appear as a different user on the host. The fix is to chmod 777 the NFS subdirectories (media, export, consume). This is fine because NFS access is already controlled at the network level by the share configuration.
The Paperless compose just points at the bind-mounted paths:
\nvolumes:\n # data stays local (search index, classifier cache)\n - /opt/paperless/data:/usr/src/paperless/data\n # media/export/consume on NFS (TrueNAS → Proxmox → LXC bind mount)\n - /mnt/nas/paperless/media:/usr/src/paperless/media\n - /mnt/nas/paperless/export:/usr/src/paperless/export\n - /mnt/nas/paperless/consume:/usr/src/paperless/consume\nData stays local for performance (search index, classifier cache). The actual documents live on the NAS.
\nEverything follows my standard homelab convention: each service gets its own directory under docker/, its own docker-compose.yml, a .env.example with placeholders, and real secrets in .env on the host (never committed).
services:\n paperless:\n image: ghcr.io/paperless-ngx/paperless-ngx:latest\n container_name: paperless\n restart: unless-stopped\n network_mode: host\n depends_on:\n - gotenberg\n - tika\n volumes:\n - /opt/paperless/data:/usr/src/paperless/data\n - /mnt/nas/paperless/media:/usr/src/paperless/media\n - /mnt/nas/paperless/export:/usr/src/paperless/export\n - /mnt/nas/paperless/consume:/usr/src/paperless/consume\n env_file: .env\n environment:\n PAPERLESS_REDIS: redis://localhost:6379\n PAPERLESS_DBHOST: localhost\n PAPERLESS_DBNAME: paperless\n PAPERLESS_DBUSER: paperless\n PAPERLESS_TIME_ZONE: America/New_York\n PAPERLESS_OCR_LANGUAGE: eng\n PAPERLESS_URL: https://paperless.localdomain\n PAPERLESS_PORT: 28981\n PAPERLESS_TIKA_ENABLED: 1\n PAPERLESS_TIKA_GOTENBERG_ENDPOINT: http://localhost:3100\n PAPERLESS_TIKA_ENDPOINT: http://localhost:9998\n PAPERLESS_TASK_WORKERS: 5\n healthcheck:\n test: [\"CMD\", \"curl\", \"-fs\", \"-S\", \"-L\", \"--max-time\", \"2\", \"http://localhost:28981\"]\n interval: 30s\n timeout: 10s\n retries: 5\n\n gotenberg:\n image: docker.io/gotenberg/gotenberg:8.27\n container_name: gotenberg\n restart: unless-stopped\n network_mode: host\n command:\n - \"gotenberg\"\n - \"--chromium-disable-javascript=true\"\n - \"--chromium-allow-list=file:///tmp/.*\"\n - \"--api-port=3100\"\n environment:\n TZ: America/New_York\n\n tika:\n image: docker.io/apache/tika:latest\n container_name: tika\n restart: unless-stopped\n network_mode: host\n environment:\n TZ: America/New_York\nGotenberg runs on port 3100 (not the default 3000) to avoid a conflict with another service.
\nservices:\n paperless-ai:\n image: clusterzx/paperless-ai:latest\n container_name: paperless-ai\n restart: unless-stopped\n ports:\n - \"3003:3000\"\n cap_drop:\n - ALL\n security_opt:\n - no-new-privileges:true\n env_file: .env\n environment:\n TZ: America/New_York\n PAPERLESS_AI_PORT: 3000\n PAPERLESS_API_URL: http://192.168.1.2:28981/api\n PAPERLESS_URL: http://192.168.1.2:28981\n AI_PROVIDER: ollama\n OLLAMA_API_URL: http://192.168.1.53:11434\n OLLAMA_MODEL: qwen3:8b\n RAG_SERVICE_URL: http://192.168.1.2:28981\n RAG_SERVICE_ENABLED: \"true\"\n SCAN_INTERVAL: \"*/2 * * * *\"\n volumes:\n - /opt/paperless-ai/data:/app/data\nThe scan interval is 2 minutes. Every 2 minutes, paperless-ai checks for new documents and runs them through the LLM. Initially I had it at 30 minutes (the default) and kept wondering why nothing was happening after uploads.
\nservices:\n paperless-gpt:\n image: icereed/paperless-gpt:latest\n container_name: paperless-gpt\n restart: unless-stopped\n ports:\n - \"3002:8080\"\n env_file: .env\n environment:\n TZ: America/New_York\n PAPERLESS_BASE_URL: http://192.168.1.2:28981\n LLM_PROVIDER: ollama\n LLM_MODEL: llama3.2:3b\n OLLAMA_HOST: http://192.168.1.53:11434\n OLLAMA_CONTEXT_LENGTH: \"8192\"\n TOKEN_LIMIT: \"1000\"\n LLM_LANGUAGE: English\n OCR_PROVIDER: llm\n VISION_LLM_PROVIDER: ollama\n VISION_LLM_MODEL: minicpm-v:8b\n AUTO_OCR_TAG: paperless-gpt-ocr-auto\n AUTO_TAG: paperless-gpt-auto\n MANUAL_TAG: paperless-gpt-manual\n PDF_OCR_TAGGING: \"true\"\n PDF_OCR_COMPLETE_TAG: paperless-gpt-ocr-complete\n PDF_UPLOAD: \"false\"\n LOG_LEVEL: DEBUG\n volumes:\n - /opt/paperless-gpt/prompts:/app/prompts\nBoth paperless-ai and paperless-gpt use bridge networking with port mapping instead of host networking. This was necessary because their default ports (3000 and 8080) were already taken by other services on the Portainer host. Neither port is configurable via environment variable, which I discovered after watching them crash-loop with EADDRINUSE errors.
This is where I spent more time than I expected. The default paperless-ai prompt does a decent job at generating titles and detecting correspondents, but it was ignoring my specific requirements: tagging documents as \"tax-deductible\" or \"medical.\"
\nThe first attempt was a long, detailed prompt with all the rules. The small model (llama3.2:3b) would fill all 4 tag slots with descriptive tags like \"Web Hosting\" and \"Invoice\" and leave no room for \"tax-deductible.\" Upgrading to qwen3:8b helped, but the real fix was restructuring the prompt.
\nWhat worked: making the classification tags mandatory and putting them before the descriptive tags. Instead of \"include the tag tax-deductible if applicable\" buried in a paragraph, I made it:
\nTag rules (CRITICAL — follow exactly):\n- Start with mandatory tags, then add up to 3 descriptive tags.\n- MANDATORY: If the expense is potentially tax-deductible (...), you MUST include the tag \"tax-deductible\".\n- MANDATORY: If the document is healthcare-related (...), you MUST include the tag \"medical\".\n- MANDATORY: If the document is a personal identity or life document, you MUST include the tag \"personal documents\".\n- Then add up to 3 descriptive tags for the document topic.\nThe word \"MANDATORY\" and \"MUST\" in caps, plus front-loading the rules before the descriptive tags, made the model consistently apply them. Small models respond better to structure and emphasis than to nuance.
\nI also found that giving the model explicit examples of what qualifies as tax-deductible (business expenses, software subscriptions, tolls, web hosting, domain renewals) produced much better results than just saying \"business expenses.\" The model doesn't have your tax accountant's intuition. Spell it out.
\nThe M4 Max handles this workload comfortably:
\n| Model | \nSpeed | \nUse Case | \n
|---|---|---|
| qwen3:8b | \n~41 tokens/s generation | \npaperless-ai (text classification) | \n
| minicpm-v:8b | \n~48 tokens/s generation | \npaperless-gpt (vision OCR) | \n
First inference after model load takes ~20 seconds (cold load into GPU memory), but subsequent requests are near-instant because OLLAMA_KEEP_ALIVE=15m keeps the model warm. For document processing (not real-time chat), this is more than fast enough.
On the Paperless side, setting PAPERLESS_TASK_WORKERS=5 lets it OCR multiple documents in parallel. HEIC photos from an iPhone take a couple of minutes each through Tesseract, so parallelism helps when you batch-upload a stack of receipts.
This project changed how I interact with paper documents. I take a photo, drag it into Paperless, and within a couple of minutes it's filed, tagged, searchable, and backed up on the NAS. Tax season next year should be a filter query instead of a camera roll excavation.
\nThe biggest lesson: Ollama on Apple Silicon is genuinely good. Something that technically works but isn't practical, claude, chatgpt and other models are far more superior. The M4 Max runs these models fast enough that document processing feels near-instant. The unified memory architecture means even the 8B parameter models load without drama. If you have a Mac collecting dust, this is a great use for it.
\nThe second lesson: prompt engineering matters more than model size. Switching from a 3B to an 8B model helped, but restructuring the prompt to front-load mandatory rules was the bigger improvement. Small local models are capable, but you have to be explicit. They don't infer what you want from context the way larger cloud models do.
\nThe third lesson: LXC containers and NFS don't mix directly. If you're running Docker inside an unprivileged LXC on Proxmox and you need NAS storage, mount on the Proxmox host and bind-mount into the LXC. Don't spend an hour tweaking NFS export settings on TrueNAS like I did. The mount() syscall is blocked inside the container. No amount of NFS flags will fix a kernel-level restriction.
Right now, the AI tags documents as \"tax-deductible\" or not. That's already a huge win. But \"tax-deductible\" is a broad bucket. When my wife's CPA asks for expenses broken down by category, we still have to sort through the pile manually.
\nIf you're wondering why doesn't she just use quickbooks or whatever. She started last year but there is still figuring out on her, what works best for her expense-wise. Sometimes she ends up paying business expense with personal card or cash, and it's important to categorize those correctly. I would say having quickbooks makes this 80% easier than previous few years. Even tho remaining 20% isn't as big it's definitely the most annoying.
\nThe v2 vision is a second step in the pipeline that takes everything tagged \"tax-deductible\" and classifies it into one of the actual IRS expense categories we use:
\n| Category | \nCategory | \nCategory | \n
|---|---|---|
| Advertising | \nMileage | \nSunpass | \n
| Parking & Tolls | \nBank Charges | \nComputer & Internet | \n
| Client Reimbursement | \nDues & Subs | \nEducation/Coach | \n
| Gifts | \nInsurance (Health) | \nInsurance (Other) | \n
| Interest | \nLegal & Prof. | \nMeals & Ent. | \n
| Office Exp. | \nPostage | \nAssistant | \n
| Repairs & Maint. | \nSupplies | \nTaxes & Licenses | \n
| Telephone | \nTravel | \nStaging | \n
| Utilities | \nPension Exp (SEP/SIMPLE) | \nPhotography | \n
| Officer's Salary | \nOther Wages | \nPayroll Taxes | \n
These are the exact column headers from the spreadsheet we use today. The dream is to upload a receipt, have the AI tag it as tax-deductible, then automatically sub-categorize it into \"Computer & Internet\" or \"Meals & Ent.\" or \"Travel.\" At year end, export a filtered view grouped by category and hand it straight to the CPA. No more multi-day spreadsheet sessions.
\nThis is probably a custom field or a second tag layer in paperless-ai, combined with a more specific prompt. Or maybe a small script that runs after paperless-ai processing and does a second pass with a more focused prompt. Either way, the infrastructure is all in place now. The hard part is done. The categorization logic is just prompt engineering.
\nI used this prompt to generate the featured image. A cozy home office desk with a stack of messy paper documents on one side being fed into a glowing portal, emerging on the other side as neatly organized digital files floating in a holographic display. A small robot assistant with a magnifying glass is examining each document. The desk has a Mac Mini and a NAS box with blinking lights in the background. Warm lighting, isometric perspective, clean vector illustration style with a slight retro-tech aesthetic.
\n","comment_id":"69e239d43cad350001301c89","feature_image":"https://emir.fyi/content/images/2026/04/ChatGPT-Image-Apr-17--2026--01_37_11-PM.png","featured":false,"visibility":"public","created_at":"2026-04-17T09:47:00.000-04:00","updated_at":"2026-04-17T13:42:11.000-04:00","published_at":"2026-01-28T15:38:00.000-05:00","custom_excerpt":null,"codeinjection_head":null,"codeinjection_foot":null,"custom_template":null,"canonical_url":null,"url":"https://emir.fyi/turning-paperless-ngx-into-a-smart-filing-cabinet-with-local-ai-and-nas-storage/","excerpt":"The Itch\n\n\nI had Paperless-ngx running. Technically. It was deployed on Portainer, had a DNS entry, a TLS cert, the whole nine yards. It also had zero documents in it. It was a ghost town with a login page. Every time I'd get a medical bill, a receipt, or a tax document, I'd do what every responsible adult does: take a photo on my phone and forget which album it ended up in.\n\n\nThe straw that broke it was tax season. Every year, my wife and I spend hours across multiple days going through all of ","reading_time":11,"access":true,"comments":false,"og_image":null,"og_title":null,"og_description":null,"twitter_image":null,"twitter_title":null,"twitter_description":null,"meta_title":null,"meta_description":null,"email_subject":null,"frontmatter":null,"feature_image_alt":null,"feature_image_caption":null}],"meta":{"pagination":{"page":1,"limit":100,"pages":1,"total":22,"next":null,"prev":null}}}