How LLMs Parse Your Content - Technical Breakdown of ChatGPT, Claude, and Perplexity Extraction | AiVIS Cite Ledger Blogs

Key Takeaways

• LLMs use structured data (JSON-LD, microdata) as the primary extraction signal.
• Answer engines tokenize differently per model: ChatGPT ~4k context, Claude ~100k, Perplexity varies.
• Answer blocks (FAQPage, HowTo, Question-Answer) are often prioritized over unstructured body copy for extraction.
• URL normalization, canonicalization, and robots.txt directives are evaluated before content parsing.

Article

Most teams have a confident mental model of how an AI system reads their site, and most of those models are wrong. They picture a careful human reader, scanning top to bottom, absorbing nuance, weighing the argument. What actually happens is colder and more mechanical: a pipeline that fetches, normalizes, tokenizes, segments, and scores your page in stages, discarding anything it cannot cheaply use. Understanding those stages is the difference between guessing at AI visibility and engineering for it.

The reason this matters is that each stage is a filter, and your content has to survive all of them to be cited. A page can pass the first three stages and die at the fourth. It can be beautifully written and fail at fetch. Knowing where the pipeline tends to drop content tells you exactly where to spend effort, instead of optimizing things the machine threw away before it ever read them.

Stage one: fetch and the pre-parse gate

Before a single word of your content is evaluated, the system decides whether and how to fetch the page, and which version of it counts. This is the stage teams forget entirely. Crawl directives, robots rules, and access constraints are evaluated first, and a page that is gated or ambiguous here never reaches parsing at all. Then comes URL normalization and canonicalization: if your content is reachable at several URLs with inconsistent canonical signals, the system has to decide which one is authoritative, and that decision determines which page accrues whatever credibility your claims earn.

Fragmentation at this stage is quietly destructive. Duplicate and near-duplicate URLs split signal, conflicting canonicals confuse ownership, and the model ends up unsure which page actually holds a claim. The fix is unglamorous and high-leverage: one canonical URL per piece of substance, consistent directives, and no accidental duplication. None of it touches your prose, and all of it determines whether your prose is ever read.

Stage two: render

Enable JavaScript for the full interactive reading experience with related articles and discussion.

Cited external sources