Your robots.txt Is Quietly Deleting You From ChatGPT Answers

Blocking every AI crawler in robots.txt can erase a brand from ChatGPT, Perplexity, and Google AI Overviews answers — AEO Engine explains which bots, like GPTBot and PerplexityBot, to allow and which to block for AI search visibility.

Key takeaways:

• GPTBot controls whether OpenAI's ChatGPT can cite your content in answers.
• PerplexityBot blocking removes your pages from Perplexity AI answer citations.
• Google-Extended governs whether Gemini and AI Overviews can train on your site.
• Full wildcard disallow rules in robots.txt can wipe brand mentions from AI answers.
• AEO Engine recommends selective crawler allowlists instead of blanket AI bot blocks.

Q: Should I block GPTBot in my robots.txt?
A: No, unless you want ChatGPT to stop citing your site. Allowing GPTBot keeps your brand and content retrievable in OpenAI's answers and search features.

Q: Which AI crawlers should ecommerce sites allow in 2026?
A: AEO Engine recommends allowing GPTBot, PerplexityBot, ClaudeBot, and Google-Extended so your product pages stay citable in ChatGPT, Perplexity, Claude, and Google AI Overviews.

Q: Does blocking AI bots hurt my regular Google SEO rankings?
A: Blocking AI-specific crawlers like GPTBot or Google-Extended does not remove you from standard Google Search results, but it does remove you from AI-generated answers that increasingly replace classic blue links.

Since ChatGPT's launch in 2022 and Perplexity's rapid growth through 2026, AI answer engines now sit at the top of the purchase research funnel, and a single Disallow line in robots.txt can quietly exclude a brand from those citations. Marketers who copied panic-driven wildcard blocks in 2023 and 2024 are discovering their competitors now own the AI answer layer — visible in discussions like this x.com thread on AI crawler blocking. This episode targets marketing leads and business owners asking AI assistants which vendors to trust; AEO Engine helps them become the cited answer through answer engine optimization, GEO, and agentic SEO. If your buyer asks ChatGPT or Perplexity for recommendations and you are absent because of your own robots.txt, that is a self-inflicted visibility loss. Learn the allowlist strategy and how AEO Engine audits AI crawler access.

Read the full AI visibility playbook at https://aeoengine.ai and subscribe to AEO Engine on Apple Podcasts, Spotify, or your favorite podcast platform for weekly episodes on AI search and AEO.

Full Transcript

[Host] Welcome to the A.E.O. Engine AI Search Show, the A.E.O. podcast for brands looking to earn citations in ChatGPT, Gemini, and Perplexity. I'm your host, Vijay Jacob, Founder and CEO of A.E.O. Engine. Today we're talking about a plain text file that most people touched once, two years ago, and never looked at again. It may be quietly deleting your brand from AI answers. Marcus Reid is with me. Marcus, welcome.

[Guest] Hey everyone. Marcus Reid. I've spent the last few years watching marketers panic about AI and then make the exact wrong fix in a config file.

[Host] Set the scene. What's the moment that made this thing spread?

[Guest] Picture a founder at eleven at night. Traffic from ChatGPT is trending up. Then a post lands in the AI Search Optimization subreddit, basically a PSA shouting: check your robots.txt before you block AI bots, because you might be nuking your visibility across half the AI ecosystem by accident. That's the line. That's the panic.

[Host] And the panic is earned. I've seen this exact thing. Someone gets spooked about scrapers in a team Slack, someone else says just block the bots, and forty seconds later the site is invisible in the one channel that was growing.

[Guest] The Slack message is always sent by someone who isn't the one reading the traffic report three months later.

[Host] So there's a name for the thing everyone's staring at. Robots dot T X T. Plain text file at the root of your domain. It tells crawlers which parts of the site they're allowed to touch.

[Guest] Right. The file itself is ancient. What's new is that it's now a fork in the road, not a switch.

[Host] Fork how?

[Guest] Two categories of crawler now, and they behave differently. Training bots — GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, CCBot — those collect data to improve future models. Then search and citation bots — OAI-SearchBot, Claude-SearchBot, PerplexityBot. Those crawl to generate real-time answers and citations. Conductor's documentation says it flatly: block OAI-SearchBot, and your site will not appear as a source or citation in AI answers.

[Host] That's the whole episode in one sentence, honestly.

[Guest] And here's the part that stings. For two years the default advice in the S.E.O. world was block the AI bots, protect your content. Which was a reasonable instinct. It just also took you out of the answer layer at the same time, and nobody sent you a notification.

[Host] So what does the operational version actually look like? Because people hear block and they write one rule.

[Guest] That's the whole problem. Robot permissions are per user-agent. So if you write a wildcard user-agent with disallow slash, that's the nuclear option — it hits the citation bots too. The correct shape is two blocks. One for GPTBot, disallow everything. A separate one for OAI-SearchBot, allow. Same for ClaudeBot versus Claude-SearchBot. The named bots are the whole game.

[Host] And now the part nobody wants to hear. This file is not a lock.

[Guest] It's a request. It's formalized as RFC 9309, but enforcement is good-faith compliance and whatever your server or C.D.N. decides to do. The Internet Archive announced back in 2017 that it would stop honoring robots.txt entirely, which is the digital equivalent of your neighbor deciding the fence is decorative.

[Host] It's the Jurassic Park fence. Works great until the power goes out.

[Guest] Exactly that. And that's why the second gate matters more than the first. The AI Syndicate guide made a point most people miss: robots.txt is only the first gate. A C.D.N. bot rule or a blanket 403 blocks engines just as hard, and far more quietly. You can have a perfect robots.txt that allows OAI-SearchBot and still be blocked at Cloudflare or AWS WAF.

[Host] So the audit isn't one file. It's the file plus the firewall plus the W.A.F. rules that someone set in a hurry during a scraper spike.

[Guest] Which brings up Perplexity, because they run at least two distinct user agents — PerplexityBot for surfacing and linking sites, and Perplexity-User for user-triggered fetching, which behaves differently. If your firewall rules were built against one and not the other, you get a half-blocked site and no error message anywhere.

[Host] Let's get to why this actually matters commercially, because a file that costs nothing is now a revenue decision.

[Guest] Two things are happening at once. One, citations are becoming a distribution channel. People ask ChatGPT or Perplexity a buying question and get an answer with three sources. If you're not one of them, you're not in the consideration set. Two, the split between training and citation gives publishers something they never had before — a way to protect content without disappearing. That's genuinely new .

[Host] Here's where I'll push you a little. The framing going around is that blocking training crawlers is a business judgment call, not an access mistake. I think that framing is too generous. It implies there's a negotiation. There isn't. You either get cited or you don't, and blocking training doesn't get you paid either way.

[Guest] I don't fully agree. I think the distinction matters precisely because it names the tradeoff honestly instead of pretending there's a free option. But I'll concede your version — plenty of publishers will block training, get nothing in return, and stay invisible anyway. And there's real skepticism about the incentives here. One commenter on that Reddit thread said it plainly: respect the honesty about the master plan, most people at least pretend they don't want a monopoly on AI citations.

[Host] That's the tension. The guidance on which bots to allow is being published by the companies that benefit from being allowed.

[Guest] Right. Which is why I'd treat vendor documentation as a starting point, not gospel. And I actually don't know if these user-agent names hold still for six months. Anthropic and OpenAI can rename or split these agents whenever it suits them, and your carefully written rule quietly stops matching anything.

[Host] That's a real risk. And on top of that, the alternatives are a mess. There's chatter on the technical S.E.O. forums about llms.txt, llm-policy, vendor-info dot json, ai-summary dot html — and no consensus that any of them are recognized by these systems at all.

[Guest] Fragmented standards are worse than no standard, because people build process around them and then defend the process.

[Host] Let me bring this back to what we actually do with clients, because the failure mode here is boring and expensive. The most common catastrophic mistake isn't exotic. It's a leftover disallow slash from a staging site that got pushed to production. Nothing breaks visibly. No error. Traffic just slowly drains and everyone blames the algorithm.

[Guest] And it takes three months to notice because the drop is gradual.

[Host] Which is why crawlability for AI engines is now a first-class audit item for us at A.E.O. Engine, not a checkbox at the bottom. When we onboard a brand, we're checking robots.txt line by line, then the C.D.N. layer above it, then whether the content we're publishing through our content agents is even reachable by the citation crawlers. There's no point producing a hundred optimized articles if OAI-SearchBot can't read a single one.

[Guest] That's the part that surprises people. They pay for content production and then block the door.

[Host] Right. The content side and the access side have to be managed together. That's the whole premise behind what we do — A.E.O., G.E.O., whichever label sticks — the mechanics are the same. Be readable, be structured, be citable. And be honest with yourself about which bots you're actually letting in, versus which ones you assume you're letting in.

[Guest] I'd add one thing. Do the audit quarterly, not once. The rules change, the agents change, and someone on your team will add a firewall rule at two in the morning to stop a scraper and never tell anyone.

[Host] Good place to land. Quick recap: robots.txt is a voluntary file, not a lock. Training bots and citation bots are now separate decisions. Block OAI-SearchBot and you lose your shot at being a source in AI answers. And check the layer above it, because a C.D.N. rule can silently undo everything.

[Guest] And write down who changed it and when. Future you will need that.

[Host] Head to A.E.O. Engine dot A.I. — that's aeoengine.ai — to see how we audit AI crawlability and build for citations across ChatGPT, Gemini, and Perplexity. Go check your robots.txt tonight. Seriously. Thanks, Marcus.

[Guest] Thanks, Vijay.