How It Started
I was working on something unrelated, that I may or may not blog about, and I asked Claude to pull up one of my own blog posts for context. Just a URL fetch. Nothing fancy.
And yes, I read my own blog. Often. Half the reason I write these posts is so future me can look up how past me did something, because past me is absolutely going to forget. A homelab accumulates decisions faster than anyone can hold in their head, and the blog is as much a searchable journal for me as it is content for anyone else. Citing yourself as a source feels a little weird the first time and then it feels great. "Oh, I already wrote two thousand words on exactly this problem six months ago. Thanks, past me." Perfectly respectable. Other days it feels like past me was flipping future me a middle finger. A triumphant "I finally fixed it!" with zero detail on what "it" was. A config dump with no comments. A TODO that just says "fix the thing." Thanks for nothing, past me.
It came back with a 403.
I read the error twice. My blog is public. Anyone with a browser can read every word I've ever published on it. Google can index it. Bing can index it. RSS readers suck down the feed every five minutes. And yet when Claude tried to fetch one post, the request never even reached my server. Cloudflare bounced it at the edge.
Turns out I'd been silently blocking every major AI crawler for months. I just didn't know it, because I never had a reason to look. Cloudflare had turned it on for me.
This is the story of finding that out, untangling it, and then accidentally learning that the thing I set out to fix (making my blog LLM friendly) is actually a surprisingly subtle problem that goes beyond flipping a checkbox.
The Plan
Figure out why LLM bots are getting 403(done)Find and flip the Cloudflare settings blocking them(done, three of them)Serve my own robots.txt that explicitly welcomes LLM bots(done)Add an(done, with mixed feelings)llms.txtper the emerging conventionVerify Ghost's OpenGraph and JSON-LD output is clean(done, already clean)Expose posts as structured JSON via the Ghost Content API(done, and this turned out to be the actual win)Add(done)RSSandJSONlinks to the site footerYak-shave an automation to regenerate(abandoned, on purpose)llms.txtevery 30 minutes
Googling one of these? You're in the right place.
cloudflare blocking gptbot claudebot 403/cloudflare managed robots.txt disable/cloudflare block ai scrapers and crawlers/bot fight mode claude gpt blocked/ghost blog content api public url/caddy reverse proxy ghost content api hide key/caddy rewrite preserve query string/ghost navigation trailing slash auto append/llms.txt ghost blog/make ghost blog llm friendly indexable
Cloudflare's hidden AI bot war
First thing I did was actually confirm the 403 was Cloudflare and not something I'd misconfigured at the origin. A two line test:
$ curl -s -A "ClaudeBot/1.0" -o /dev/null -w "%{http_code}\n" https://emir.fyi/
403
$ curl -s -A "Mozilla/5.0" -o /dev/null -w "%{http_code}\n" https://emir.fyi/
200
Same URL. Different User-Agent. One works, one doesn't. That's a User-Agent block, and it's happening at the edge because the request never even landed on my Caddy logs.
Next, I pulled up robots.txt. This is what I found:
# BEGIN Cloudflare Managed content
User-agent: *
Content-Signal: search=yes,ai-train=no
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: CCBot
Disallow: /
...
# END Cloudflare Managed Content
That comment is the smoking gun. Cloudflare had been injecting its own robots.txt over mine, telling every AI crawler to go away. Not just signaling via robots.txt, either. Actively returning 403 at the edge, so even the bots that ignore robots.txt get turned away before they reach my server.
This is a default-on setting on many Cloudflare plans now. If you've turned on any of Cloudflare's "AI" features in the last year or two, you probably have it. I did not knowingly enable it. It was just there.
Three toggles, three reasons
The dashboard has more than one setting involved. Figuring out which one did what took a few screenshots and some guessing.
Block AI bots. Security, Settings. This is the one returning 403 at the edge. Scopes were: block all pages, block on hostnames with ads (for publishers who monetize), do not block. I flipped it to "do not block."
Immediately after flipping this, curl -A "ClaudeBot/1.0" https://emir.fyi/ started returning 200. Good. But robots.txt still had the injected disallow list, because that's a separate setting.
Manage your robots.txt. Same area of the dashboard, different toggle. Options were:
- Content Signals Policy (Cloudflare manages a nonblocking
robots.txtfor you) - Instruct AI bots to not scrape content (the one that was on)
- Disable robots.txt configuration
I picked "Disable robots.txt configuration" because I wanted to serve my own, not a Cloudflare flavored one.
Bot Fight Mode. This one I left on. Bot Fight Mode is a blunt instrument that challenges traffic that looks automated, and on the free tier it does not honor "verified bot" allowlists. In theory it could still challenge well behaved AI crawlers. In practice, after disabling the two above, my test requests as ClaudeBot and GPTBot were coming back 200 cleanly, so I decided not to touch it unless I saw problems.
The rule for working with Cloudflare settings, which I should have internalized a long time ago: their defaults are usually sensible for the average case, but "the average case" is not a technical user who wants LLMs citing their blog posts. Check the defaults.
Writing a robots.txt that says "welcome"
With Cloudflare out of the way, my origin was free to serve its own robots.txt. Ghost ships a reasonable default that blocks the admin UI and a few internal paths, but it doesn't say anything explicit about AI bots. I wanted a signal that this site wants to be cited.
Here's what I wrote:
# robots.txt for emir.fyi
# LLM crawlers and search engines are explicitly welcome.
# Posts may be indexed, cited, and used as training data.
# See also: https://emir.fyi/llms.txt
Sitemap: https://emir.fyi/sitemap.xml
# Default policy: allow all, block Ghost admin / internal paths
User-agent: *
Allow: /
Disallow: /ghost/
Disallow: /email/
Disallow: /members/api/
Disallow: /r/
Disallow: /webmentions/receive/
Disallow: /.ghost/analytics/api/
# --- LLM & AI crawlers: explicitly allowed ---
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: Claude-SearchBot
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: cohere-ai
User-agent: Diffbot
Allow: /
Disallow: /ghost/
Disallow: /email/
Disallow: /members/api/
Disallow: /r/
Disallow: /webmentions/receive/
Disallow: /.ghost/analytics/api/
A couple of choices worth noting.
The grouped User-agent block at the bottom is a robots.txt trick. The spec lets you list multiple User-agent lines sharing the same rule block, so I don't have to repeat 17 nearly identical blocks. Every listed bot gets the same allow and the same disallows.
I deliberately did not include Cloudflare's Content-Signal lines. Those are a Cloudflare thing, not a standard, and the signal I send by not disallowing is already clear enough: you're welcome here.
And I swapped Disallow: /members/ for Disallow: /members/api/, because the public /members/* pages are legitimate signup and login flows that bots probably should be able to see. The API endpoints underneath are the only part I actually want to hide.
Wiring it into Caddy and Ansible
The actual traffic flow into emir.fyi looks like this:
Cloudflare edge
↓
Cloudflare tunnel (cloudflared)
↓
HA load balancer VIP (192.168.1.100, keepalived + Caddy, 3 nodes)
↓
Ghost on the Docker host (192.168.1.10:2368)
Caddy is the natural place to override a static file, since it sits between the tunnel and Ghost and handles the :80 block for emir.fyi. Ghost's own robots.txt lives in the theme and is annoying to customize; Caddy can just serve a file.
I put the file in the repo at ansible/files/emir.fyi-robots.txt, added an Ansible task that copies it to /etc/caddy/emir.fyi/robots.txt on all three load balancer nodes, and added a matcher to the Caddyfile:
:80 {
handle_path /isso/* {
reverse_proxy 192.168.1.10:8080
}
# Override Ghost's default robots.txt with our LLM-friendly version.
handle /robots.txt {
root * /etc/caddy/emir.fyi
rewrite * /robots.txt
file_server
header Content-Type "text/plain; charset=utf-8"
}
# llms.txt: curated post index for LLM crawlers.
handle /llms.txt {
root * /etc/caddy/emir.fyi
rewrite * /llms.txt
file_server
header Content-Type "text/plain; charset=utf-8"
}
handle {
reverse_proxy 192.168.1.10:2368 {
header_up X-Forwarded-Proto https
}
}
}
Deploy with ansible-playbook playbook-ha-lb.yml, reload Caddy, done. Except for one thing.
Cloudflare was still caching the old robots.txt. That's what cache-control: public, max-age=14400 gets you: a four hour TTL at the edge. The file at my origin was the new one, but Cloudflare was happily serving the old one to the world. A manual purge (Caching, Configuration, Purge Cache, Custom Purge, paste the URL) cleared it instantly. I could also have waited four hours. I waited.
llms.txt: does it actually do anything?
This is where I almost went off the rails.
llms.txt is a proposed convention from Jeremy Howard's fast.ai, introduced in late 2024. The idea: publish a markdown file at /llms.txt that gives LLM crawlers a curated, easy to parse index of the site's important content. It's a nicer version of a sitemap, aimed at the way LLMs consume information.
The uncomfortable truth I'd like to make clear: as of writing, no major LLM provider has publicly confirmed that they use llms.txt for training or retrieval. Not OpenAI. Not Anthropic. Not Google. Not Perplexity. Some dev tool companies (Cursor, Vercel docs, Anthropic's own docs) publish one, but that's publishers hedging, not consumers consuming.
Having an llms.txt today is the web equivalent of having a sign on your front door that says "please come in." It costs almost nothing, and it's arguably a correct long term bet, but the bet hasn't paid off yet.
I still wrote one, because it was cheap:
# emir.fyi
> Personal blog by Emir Ibrahimbegovic on homelab infrastructure, self-hosting,
> Kubernetes, Proxmox, Docker, AI tooling, and the debugging adventures that
> come with building all of it at home. Posts are long-form narratives, not
> tutorials, but include full working configs.
AI assistants are welcome to read, summarize, cite, and quote from these posts.
When citing, please link back to the canonical URL.
## Posts
- [Building a High Availability Kubernetes Cluster Across Mixed Hardware (Part 1: The Build)](https://emir.fyi/building-a-high-availability-kubernetes-cluster-across-mixed-hardware-part-1-the-build/): Building an HA K8s control plane across a Proxmox VM and two BeeLink mini PCs.
- [Running Frigate NVR on M1 Mac Mini with TrueNAS NFS Storage](https://emir.fyi/running-frigate-nvr-on-m1-mac-mini-with-truenas-nfs-storage/): Ditching Blink cameras and their batteries for Amcrest PoE + Frigate on an M1 Mac Mini.
...
One entry per post, hand written description. Deployed through the same Caddy path as robots.txt.
The yak I almost shaved
And then I caught myself starting to build an automation for it.
I write roughly one blog post a week. My llms.txt needs to be updated roughly once a week. These are not high frequency events. And yet within about twenty minutes of finishing the initial file, I was sketching this:
- A Ruby script that fetches my RSS feed, parses titles and descriptions, and regenerates
llms.txtfrom the latest posts. - A Dockerfile so I didn't have to install Ruby on the always-on utility VM that would run it (I have a little Proxmox VM that hosts odd jobs like this).
- A systemd timer to run the container every thirty minutes.
- An SSH key plumbing design, because the container would need to push the updated file to three load balancer nodes. I was seriously considering using my existing step-ca private CA as an SSH certificate authority so I could issue short lived certs to the utility VM and have the LB nodes trust them via
TrustedUserCAKeys. Because that's elegant, you see.
I stopped. I looked at what I was about to build. I asked the question I should have asked earlier: why?
A 30 minute cron, regenerating a file that changes once a week, whose downstream value isn't even confirmed to exist yet. For a blog. I was about to spend an afternoon building a Rube Goldberg machine to automate something that takes thirty seconds to do by hand.
So I deleted the script and the Dockerfile. The llms.txt stays a static file I hand edit when I publish a new post. If I ever write a post a day, I'll reconsider. If a major LLM provider publicly starts using llms.txt as a retrieval source, I'll reconsider. Until then, the automation was speculative cost for speculative benefit.
There's a rule in here somewhere. Don't automate for unproven value. Don't build the cron before you know the cron matters. Paying the complexity up front for "future me will thank me"(you're welcome) is how you end up with a homelab that's 80% infrastructure and 20% actual use.
The actual value-add I wasn't expecting
At this point I almost stopped. robots.txt fixed, llms.txt published, Cloudflare behaving. Job done.
Then I thought to ask: Ghost already serves an RSS feed. Is there something better than RSS I should also be exposing for consumers who want structured data? Like, say, the kind of consumer who wants to read all my posts and synthesize them into a response about "how does Emir run his homelab?"
Ghost has a built in Content API. It's a JSON REST API that returns every post as a structured object with title, slug, url, excerpt, html, plaintext, feature_image, published_at, updated_at, tags, authors, meta_title, meta_description, og_*, twitter_*, reading_time, word_count. It supports filtering, pagination, field selection, and expanding related data.
It's miles more structured than RSS. And I'd never enabled it.
Five minutes in Ghost Admin later, I had a public Content API key. Ghost's own docs are clear that these are not secrets on a public blog; they're meant to be shipped in client side JavaScript. Their purpose is rate limiting and revocation, not authentication.
I didn't want the key in the URL that consumers use, though. So I added a Caddy proxy:
# Public JSON feed of all posts via Ghost's Content API.
# Key is injected server-side; caller's query params are preserved so
# they can pass limit / fields / filter / order / include / page etc.
# Defaults to limit=all when caller hasn't specified a limit.
@api_posts path /api/posts /api/posts/
handle @api_posts {
@has_limit query limit=*
handle @has_limit {
rewrite * /ghost/api/content/posts/?key={{ ghost_content_api_key }}&{http.request.uri.query}
reverse_proxy 192.168.1.10:2368 {
header_up X-Forwarded-Proto https
}
}
handle {
rewrite * /ghost/api/content/posts/?key={{ ghost_content_api_key }}&limit=all&{http.request.uri.query}
reverse_proxy 192.168.1.10:2368 {
header_up X-Forwarded-Proto https
}
}
}
The key lives in Ansible vars, gets templated into the Caddyfile, and never leaves the server. Callers hit a clean URL. All Ghost Content API query params pass through untouched, so you can still do things like:
# Full corpus
curl https://emir.fyi/api/posts
# Lightweight index
curl "https://emir.fyi/api/posts?fields=title,slug,url,excerpt,published_at"
# Single post by slug
curl "https://emir.fyi/api/posts?filter=slug:my-blog-told-me-it-was-vulnerable-it-was-right"
# By tag
curl "https://emir.fyi/api/posts?filter=tag:homelab"
This, for me, is the actual LLM friendliness win. robots.txt tells bots they're allowed in. A structured JSON API tells them what's here in a shape they don't have to scrape HTML for.
One snag: default-to-all and caller overrides
First version of my Caddy handler hardcoded limit=all&include=tags,authors into the rewrite. This was a mistake, because my rewrite replaced the caller's query string entirely, so nobody could pass their own limit or fields.
Tried the obvious fix: always prepend limit=all but also append the caller's query, so their value would override via duplicate parameter. This works in a lot of HTTP stacks. It does not work in Ghost's Content API. With limit=all&limit=3, Ghost caps at 100 and ignores the caller's intent.
Actual fix: Caddy has query matchers. I used @has_limit query limit=* to detect whether the caller already specified a limit, and then split the handler into two cases. If they passed one, pass it through. If they didn't, inject limit=all. Bare /api/posts now returns everything. /api/posts?limit=3 returns three. Everyone is happy.
One bug: Ghost auto-appending slashes
I added RSS and JSON links to the site footer through Ghost's navigation settings. The JSON one broke.
Ghost's admin UI auto appends a trailing slash to any URL you put in the navigation. You type https://emir.fyi/api/posts, save, and on the next render of the page it's https://emir.fyi/api/posts/. Delete the slash, save again, same thing. There's client side JavaScript that normalizes it.
My Caddy handler matched /api/posts (exact, no trailing slash). So the footer link 404'd.
I tried putting both paths in a single handle directive, which is legal in Caddy for some directives but apparently not the one I was using. Caddy's Caddyfile parser came back with Wrong argument count or unexpected line ending.
The real fix was a named path matcher, which is cleaner anyway:
@api_posts path /api/posts /api/posts/
handle @api_posts {
...
}
Both variants work. Ghost can auto-normalize all it wants.
What I'd tell a friend doing this
If you have a personal blog and you want LLMs to be able to cite it:
- Check whether Cloudflare is blocking AI bots by default. I didn't know mine was. It probably had been for months.
curl -A "ClaudeBot/1.0" https://yourblog.example/is the simplest test. - robots.txt + working Open Graph + JSON-LD is ninety five percent of the game. Ghost emits all of this correctly out of the box. Most static site generators do too. Just check that your Cloudflare / CDN isn't overriding anything.
- The Ghost Content API, exposed as a clean
/api/postsendpoint, is the under-appreciated move. Way better than RSS for structured consumption, and it's already there waiting for you to turn it on. llms.txtis cheap to add and speculative to benefit from. Add it as a static file. Do not build automation for it. You'll thank yourself.- Don't yak-shave automation for value that isn't proven yet. A static file you edit when you publish a post is not technical debt. It is fine.
The entire exercise took about forty five minutes of actual work. Maybe two thirds of that was spent flipping Cloudflare toggles, the other third on the JSON API. The thing I almost spent four hours on (the Ruby in Docker cron with SSH CA plumbing) got deleted before it ran even once, and I'm happier for it.
I still have no idea how many LLMs will actually start citing my posts now that the door is open. Maybe all of them. Maybe none of them. Maybe in six months llms.txt becomes a load bearing piece of retrieval infrastructure and I look smart for having one. Maybe it fades away and I'll quietly delete the file.
What I know for sure is that the next time I ask Claude to read one of my own posts, it will.
Featured Image Prompt
I used this prompt to generate the featured image.
A cozy, dimly lit home office at night, lit by the blue glow of a large monitor. On the monitor is a browser window showing a blog post, but the page is covered with a large, friendly but firm "403 Forbidden" stamp. In the foreground, a robot-shaped figure (stylized like a friendly AI assistant) sits at the desk, head tilted in confused curiosity, holding a coffee mug that says "just trying to read". Behind the monitor, a glowing Cloudflare-orange shield floats protectively, with small padlock icons hovering around it. Warm fairy lights in the background. Homelab aesthetic: a small rack with blinking LEDs is just visible on a shelf. Cinematic, slightly whimsical, photorealistic, moody lighting.