Rotating WireGuard Keys After I Committed Them to Git

Why I'm Writing This One

Most of my blog posts are about building something new. Setting up a service, wiring things together, getting the green light. Those are fun to write. But I think there is more to learn from the posts where something went wrong.

Mistakes in infrastructure are not rare. They are inevitable. What matters is how quickly you catch them and how cleanly you recover. This post is about a time I committed WireGuard private keys to a Git repo and had to rotate everything. It is not a proud moment, but it is an honest one, and the cleanup turned out to be surprisingly clean because of decisions I had made months earlier without thinking about this exact scenario.

Starting from a good position made all the difference. The repo was private, so the leak's blast radius was limited to me and GitHub's servers, not the entire internet. I already had .gitignore patterns covering other secrets (.env files, Terraform tfvars, Ansible vault keys), so the fix was extending an existing pattern rather than inventing one from scratch. The WireGuard setup was fully automated with Ansible, which meant key rotation was a scriptable operation, not a manual slog through config files on three devices. And because I am the only person pushing to this repo, a history rewrite with git filter-repo plus a force push was safe with zero coordination overhead.

None of those things were set up in anticipation of this specific mistake. They were just good defaults that happened to pay off when I needed them. The lesson, if there is one before the story even starts, is that good hygiene compounds. The day you need it, you will be glad you bothered.

How It Started

I was adding my wife's laptop to the WireGuard VPN so she could reach Fizzy, our family project management tool, from anywhere. The add-client Ansible playbook spits out a config file and drops it in the repo under ansible/wireguard-wifes-laptop.conf. I committed it, was about to push, and then paused. Something nagged at me.

I opened the file to double-check what was in it.

[Interface]
PrivateKey = AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
Address = 10.200.200.3/32
DNS = 192.168.1.1

[Peer]
PublicKey = BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB=
Endpoint = vpn.example.com:51820
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 25

That PrivateKey line is exactly what it looks like. It is the private half of a WireGuard keypair, freshly generated and sitting one git push away from being on GitHub forever. Worse, I realized my "add-client" playbook had been fetching configs into the repo directory since the day I wrote it, and I'd already shipped two previous clients the exact same way.

I opened git log. Yep. Commits titled "my laptop" and "wireguard, blogs and fizzy" from a couple weeks back. Private keys for my my laptop and my phone, sitting in the repo, pushed to origin.

The repo is private, so only GitHub and I have seen them. But "private" on GitHub means "private from the internet," not "never left my computer." Those keys were backed up, replicated, and indexed on somebody else's servers. They are, in the most literal sense, compromised.

The Plan

This is not a "run one command and fix it" problem. It needs a sequence, in this order:

1. Stop the bleeding. Get the new config out of the index so it does not ship.
2. Gitignore going forward. No future .conf in the repo, ever.
3. Rotate the two compromised peers. New keypairs on the server, new configs on the devices.
4. Remove the tracked files from HEAD. Clean current state of the repo.
5. Audit the rest of the repo. If I made this mistake once, I probably made it somewhere else.
6. Decide about history. The bad keys are still in old commits. Do I care?

I ran git reset HEAD~1 to undo the commit before anything else. That bought me time to think.

Stopping the Bleeding

The reset dropped the wifes-laptop .conf back to untracked. I added a pattern to .gitignore that blocks the whole class of file:

# Secrets
ansible/wireguard-*.conf
!ansible/wireguard-*.conf.example

The ! exception lets me keep committing .example templates, which is how every other secrets file in this repo already works. I verified the pattern with git check-ignore -v ansible/wireguard-wifes-laptop.conf and got back the exact rule that matched. Good.

Important caveat. .gitignore does not untrack files that are already tracked. The two already-committed configs were still in the index. They would stay in the repo until I explicitly removed them, which was a later step.

Checking the Server Before Touching Anything

Before rotating keys I wanted to see what state the WireGuard server was actually in. You cannot safely rotate peers if you do not know what peers exist.

ubuntu@vpn:~$ sudo wg show
interface: wg0
  public key: BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB=
  listening port: 51820

peer: CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC=
  allowed ips: 10.200.200.2/32
  latest handshake: 12 days ago

peer: DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD=
  allowed ips: 10.200.200.2/32
  latest handshake: 1 day ago
  transfer: 3.14 GiB received, 23.33 GiB sent

peer: EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE=
  allowed ips: 10.200.200.3/32

Three peers. Pay attention to the allowed ips column. Two peers both claim 10.200.200.2/32. That is not supposed to happen.

The bootstrap playbook generates a client config during initial setup. The add-client playbook adds a second client but counts existing peers by listing directories under /etc/wireguard/clients/. Since the bootstrap client was not in that directory, the add-client logic thought there were zero clients, so it gave the new peer IP .2. Same IP as the bootstrap peer. WireGuard tolerated the collision by routing whichever peer had handshaked most recently.

That explains why my phone had felt flaky for the past couple weeks. The my laptop, handshaking more often, was silently stealing the .2 slot.

Two bugs for the price of one.

The Surgical Rotation

The naive approach would be to rotate every peer, restart the whole interface, and move on. I did not want to do that. My wife's laptop peer was not compromised. Dropping her tunnel because I was cleaning up my mess would be rude.

The right tool is wg syncconf, which applies a config file to a running interface without touching peers that did not change. The invocation is a bit awkward because you have to strip comments and [Interface] post-up lines that wg itself cannot parse:

sudo wg syncconf wg0 <(wg-quick strip wg0)

My plan was:

1. Back up wg0.conf with a timestamp.
2. Remove the two compromised peers from the running interface with wg set wg0 peer <pubkey> remove. This drops them immediately so the leaked keys become useless.
3. Rewrite wg0.conf from scratch containing only the Interface section, my wife's unchanged peer block, and two fresh peer blocks with newly generated pubkeys.
4. Apply with wg syncconf.
5. Emit new client configs on the server for me to fetch.

Because I was doing all of this on one host, I bundled it into a single bash script and piped it over SSH. Half the complexity of a rotation like this is "did I miss a step and now half the interface is broken." Running everything in one atomic-ish script means you either end in a good state or you restore from backup.

The keygen loop looks like this:

PHONE_PRIV=$(wg genkey)
PHONE_PUB=$(echo "$PHONE_PRIV" | wg pubkey)
printf '%s' "$PHONE_PRIV" > /etc/wireguard/clients/phone/private.key
printf '%s' "$PHONE_PUB"  > /etc/wireguard/clients/phone/public.key

Then the rewritten wg0.conf:

[Interface]
PrivateKey = FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF=
Address = 10.200.200.1/24
ListenPort = 51820
PostUp = iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

# BEGIN CLIENT wifes-laptop
[Peer]
# wifes-laptop
PublicKey = EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE=
AllowedIPs = 10.200.200.3/32
# END CLIENT wifes-laptop

# BEGIN CLIENT phone
[Peer]
# phone
PublicKey = GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG=
AllowedIPs = 10.200.200.2/32
# END CLIENT phone

# BEGIN CLIENT my-laptop
[Peer]
# my-laptop
PublicKey = HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH=
AllowedIPs = 10.200.200.4/32
# END CLIENT my-laptop

I gave the my laptop .4 on purpose. Previously it was colliding with the phone at .2. By spreading them out I fixed both the leak and the bug.

The script also emitted a client .conf for each device and, for the phone, ran qrencode -t png -o /tmp/phone.png so I could scan it with the WireGuard mobile app. The laptop got a plain .conf file, fetched with scp into the gitignored path in my local repo.

After wg syncconf, the server showed three peers with brand new public keys. My wife's tunnel never blipped.

Auditing the Rest of the Repo

If I made this mistake with WireGuard, what other secrets might be sitting in tracked files?

I did a grep sweep across all 57 tracked files, looking for:

• Private keys of any kind (BEGIN PRIVATE KEY, PrivateKey =, raw key files).
• API tokens (Cloudflare, Tailscale, GitHub, etc.) with their usual prefixes.
• .env files (should be gitignored, are any slipping through?).
• Unencrypted sealed-secret YAML sources.
• Terraform tfvars with real values.
• Hardcoded passwords in docker-compose files.
• Blog posts with real IPs, hostnames, or credentials in code blocks.

The audit came back clean. Every PrivateKey = occurrence in tracked files was either a Jinja template variable like {{ wg_client_privkey.content | b64decode | trim }} or an explicit docs placeholder like <server_private_key>. No tokens. No tfvars. Blog posts were already using placeholder IPs per my own style rules.

That was a genuine relief. The WireGuard slip was the only one.

The History Rewrite Question

Rotating keys kills the leaked material at its point of use. It does not remove those keys from commit history. Old commits on origin still had the private keys in them. What to do about that is a judgment call.

Four options, ranked by effort:

I picked option two. Private repo, solo maintainer, keys already rotated. The only value left in scrubbing history was "I will sleep slightly better," which is not nothing.

Running git filter-repo

First, install it. brew install git-filter-repo.

Before doing anything destructive, I bundled the entire repo as a backup. If I screwed up the rewrite I could restore from this file in one command.

git bundle create /tmp/homelab-prefilter-backup.bundle --all
git bundle verify /tmp/homelab-prefilter-backup.bundle

Then the rewrite itself:

git filter-repo --invert-paths \
  --path ansible/wireguard-client.conf \
  --path ansible/wireguard-my-laptop.conf \
  --force

--invert-paths means "keep everything EXCEPT these paths." Without it, filter-repo keeps only what you name.

One surprise. git filter-repo deliberately removes the origin remote after rewriting, so you cannot accidentally push the rewritten history to the wrong place. You have to re-add it manually:

git remote add origin [email protected]:user/repo.git

Before pushing I verified the rewrite:

git log --all -p | grep -E 'leaked-key-fragment-1|leaked-key-fragment-2'

Zero hits. The two specific leaked private keys were gone from every commit on every branch. I also did a byte-level blob hash comparison between the pre-filter and post-filter file trees. Every shared filename had identical content, with one expected exception: .gitignore, which I had updated as part of the fix.

Time to push.

git push --force origin main

+ c1246f9...278fc57 main -> main (forced update)

Done.

The Plot Twist: Stale Branches

While preparing the push I noticed the backup bundle had references to nine other branches on origin. Feature branches from old work: add-beszel-monitoring, ha-lb, paperless-real-setup, and so on. Things I had merged and forgotten about.

This was worth checking. git filter-repo rewrote every local ref, but those remote branches on GitHub still had their old SHAs. If any of them contained the leaked .conf files, my force push of main would not have fixed them.

for ref in $(git for-each-ref --format='%(refname)' refs/heads/ refs/remotes/); do
  hits=$(git log "$ref" --oneline -- ansible/wireguard-client.conf ansible/wireguard-my-laptop.conf)
  [ -n "$hits" ] && echo "HITS on $ref: $hits"
done

Nothing. The leaked files had only ever touched main. Every other branch was cut from earlier in history and never saw the .conf files.

Rather than leave those zombie branches sitting on origin, I deleted them all. They were merged or abandoned anyway.

git push origin --delete \
  add-beszel-monitoring add-lb-m1-node add-paperless-and-redis \
  docs/terraform-beszel-m4-lb feature/docker-log-rotation-and-healthchecks \
  ha-lb update-docs-and-network-info

Two of the original nine did not exist on remote (only local), so I dropped them from the command and the rest deleted cleanly. git ls-remote origin now returns a single ref: main.

What I'd Change in the Playbook

The real lesson is not "do not forget to gitignore things." Humans forget. The real lesson is that any playbook generating secret material should refuse to emit that material to a path that could ever end up in version control. Defense in depth, not vigilance.

There are a few ways to fix this, each with trade-offs:

Option A: emit outside the repo. Change the fetch task to drop the client config at ~/wireguard-configs/<name>.conf. The file never lives in the repo directory, so it cannot be committed. This is the simplest option and what I plan to do.

Option B: never write it to disk. Pipe the QR code and the config contents straight to stdout, print them once, and move on. No file artifact at all. Works great for phone configs that get scanned and discarded. Less great for laptops where you want to copy a file over.

Option C: require a sentinel gitignore. Make the playbook check for ansible/.gitignore containing wireguard-*.conf before it runs. Fail loudly if the pattern is missing. This prevents the mistake but still puts the file in a dangerous place.

Option A wins on simplicity.

The IP assignment bug is a separate fix. The add-client playbook should scan wg0.conf for already-allocated AllowedIPs values and pick the lowest free address, rather than counting directories. That one goes on the backlog.

Lessons

The obvious lesson is to add the gitignore pattern before you generate the first secret. Everyone says this. Nobody does it until they have the scare.

The less obvious lesson is how much an asking-before-acting habit saved me. I was about to push my wife's config. I opened the file because something felt off. Reading the file before pushing is a five-second operation that prevented the third leak and triggered the cleanup of the first two. That pause is the entire game. Not tools, not gitignore patterns, not hooks. Just slowing down enough to look at what you are about to ship.

The best outcome of all of this is not that I cleaned up a leak. It is that I now have a WireGuard server with a correct peer table, a clean repo history, a playbook that is going to get refactored, and a visceral memory of why the refactor matters.

The phone's tunnel came back on the first scan. New IP, new key, same experience. The leaky months of flaky connectivity from the .2 collision, gone as a bonus. Infrastructure debt paid off sideways, which is my favorite kind.

Oh did I mention, I also have a githook now which tasks my local LLM to inspect what I am about to push to repo and whether it has secrets or not. I guess if you've not read this far, you will never know.

Featured Image Prompt

I used this prompt to generate the featured image.

A homelab aesthetic, an open 1U rackmount server rendered in deep blues and warm amber LEDs, with an ethereal ghost-like stream of glowing keys made of light escaping upward from a git branch rendered as a stylized circuit-board trace, the keys disintegrating into pixels as they rise. Tech-noir color palette, cinematic lighting, a faint terminal prompt glow reflecting off the server chassis. Vector-ish line work mixed with subtle 3D render.