Why retrieval and structure matter
A webpage can be clear and convincing to a person while being difficult for an automated client to process. Important copy may only appear once JavaScript has run. A firewall may treat non-browser requests as suspicious. Headings, canonical addresses and structured data may be absent. Ownership, authorship and dates may be obvious to a reader and invisible in the markup.
None of this makes a website invisible to AI on its own, and fixing it guarantees nothing. But one thing is certain: if a retrieval agent cannot fetch your page, it cannot cite it. Whether it would have cited you is the uncertain part. The retrieval is not.
It helps to think of it in three stages.
Retrieval – Can the agent reach the page and its supporting discovery files at all? This is where stated policy and actual behaviour most often diverge. A robots.txt that welcomes every crawler counts for nothing if a CDN rule written years ago refuses anything that isn’t a browser.
Extraction – Does the response contain enough useful content to work with? Not every retrieval client executes page JavaScript fully, so important content that only appears in the browser may not be recovered. A page assembled in the browser can look immaculate to you and arrive as an empty shell.
Interpretation – Does the page identify its subject, its structure, the organisation behind it and the signals that let a reader judge the source?
Each stage depends on the one before it. A beautifully structured page nobody can fetch scores nothing that matters, which is why our scoring rubric weights access most heavily.