How to Test If AI Can Extract Your Content | AiVIS.biz
You think your content is visible to AI. But have you tested it? AI crawlers see a different version of your site than your browser shows you.
View source, not inspect element
The first test is simple: right-click your page and select 'View Source' (not Inspect Element). If the HTML body is mostly empty — just a <div id='root'></div> and script tags — your site is client-rendered. AI crawlers see that empty shell, not your actual content.
Inspect Element shows the DOM after JavaScript execution. View Source shows what the server delivers. AI crawlers receive the server's response.
Check robots.txt for AI crawler blocks
Visit yoursite.com/robots.txt and search for GPTBot, ClaudeBot, PerplexityBot, Googlebot. If any of these are blocked with 'Disallow: /', that crawler cannot access any page on your site. Many security configurations add these blocks by default.
Run an extraction-focused audit
Manual checks cover the basics. For a comprehensive extraction readiness assessment, run an AiVIS.biz audit. The audit crawls your page via headless browser (simulating AI crawler behavior), evaluates all six extraction dimensions, and reports specific findings with evidence IDs.
Each finding tells you exactly what is missing and how to fix it — no guesswork, no generic advice.
Frequently Asked Questions
- Can I test for free?
- Yes. Observer tier is free forever and provides the audit score plus your top 3 blockers. Full recommendations with implementation code require Starter or above.
- How often should I test?
- Test after any structural change (new schema, robots.txt update, SSR migration). Monthly re-audits catch drift from AI model behavior changes.