Your robots.txt Is Blocking ChatGPT, Claude, and Perplexity, And You Probably Do Not Know It | AiVIS Cite Ledger Blogs

By · · 8 min read · SEO

A single Disallow: / under the wrong user-agent and your content will never enter an AI model index. Most teams never check. Here is the 30-second audit.

Key Takeaways

  • A Disallow: / under GPTBot, ClaudeBot, or PerplexityBot removes you from that platform's index entirely.
  • AI-crawler blocks usually arrive by accident via plugins, CDN/WAF toggles, or copied templates.
  • The free Robots Checker grades each crawler A–F separately and reads robots.txt, meta robots, and X-Robots-Tag.
  • An explicit Allow: / for a named bot overrides a wildcard block.
  • Technical SEO (crawl access, meta, headers) is a large share of the AiVIS citation score, analyzed by default, not a separate product.

Article

Go check your robots.txt right now. Open yourdomain.com/robots.txt in a browser and look for these lines:

User-agent: GPTBot

User-agent: ClaudeBot

User-agent: PerplexityBot

If any of them is followed by Disallow: / then you have told that AI platform, in writing, not to crawl you. It will honor that. Your content will not enter its index, and it cannot be cited in answers, especially the time-sensitive ones that get regenerated from a live crawl.

This is the most common, most invisible AI-visibility failure we see. It is invisible because nothing breaks. Your rankings hold. Your analytics look normal. Google still sends traffic. But the moment a customer asks ChatGPT, Claude, or Perplexity about the problem you solve, you are simply not in the candidate set.

Why this happens by accident

Almost nobody sits down and decides to block AI crawlers. It happens three ways:

1. A security or "bot management" plugin adds blanket AI-crawler blocks in an update.

2. A CDN or WAF rule (Cloudflare, for example) flips on an "AI Scrapers and Crawlers" toggle.

3. A developer copies a robots.txt template from a forum that includes AI blocks "to be safe."

The result is the same: a directive you never reviewed is quietly removing you from the answer layer.

Who these crawlers actually are

Not all AI crawlers do the same job, and that matters for what you allow:

  • GPTBot, OpenAI training and browsing for ChatGPT. Critical.
  • ChatGPT-User and OAI-SearchBot, OpenAI live browse and SearchGPT. Critical.
  • ClaudeBot and Anthropic-AI, Anthropic, for Claude. Critical.
  • Google-Extended, Google's Gemini training control. Critical.
  • Googlebot and Bingbot, classic search index, but Bingbot also feeds Copilot. Critical.
  • PerplexityBot, Perplexity's research crawler.

Blocking a critical crawler is a direct hit to whether that platform can ever cite you.

The 30-second check (free, no account)

You do not have to read raw robots.txt yourself. The free AiVIS Cite Ledge

Enable JavaScript for the full interactive reading experience with related articles and discussion.