Why Claude Can't Find My Website | AiVIS Cite Ledger

Claude (by Anthropic) combines training data with real-time web access via ClaudeBot. If Claude gives wrong or missing information about your site, the extraction pipeline is failing.

How Claude accesses web content

Claude uses two pathways: pre-trained knowledge (from its training corpus) and real-time web access via ClaudeBot. The training corpus captures a snapshot of the web, if your content was not well-structured at the time of training, Claude's base knowledge may be incomplete or incorrect.

ClaudeBot respects robots.txt directives. If your robots.txt blocks Anthropic's crawler (User-agent: ClaudeBot or User-agent: anthropic-ai), Claude cannot access your content for real-time responses.

Common reasons Claude misses your site

ClaudeBot blocked in robots.txt. Your site was poorly structured at training time and no fresh crawl updates have occurred. Client-side rendering delivers empty HTML to ClaudeBot. Missing entity metadata prevents Claude from attributing content to your domain.

Fixing Claude access

Allow ClaudeBot in robots.txt. Ensure server-side rendering. Add Organization and Article JSON-LD. Write clear, attributable content. Run an AiVIS Cite Ledger audit to identify the specific gaps in your extraction readiness for Claude and other models.

Frequently Asked Questions

Does Claude use a different crawler than ChatGPT?
Yes. Claude uses ClaudeBot (or anthropic-ai), while ChatGPT uses GPTBot. Each requires separate robots.txt allowances.
Can I test what Claude knows about my site?
AiVIS Cite Ledger citation testing evaluates brand presence across multiple AI models including Claude. You can see whether Claude mentions your brand and how accurately.