Your website may have dozens of high-quality content, but the AI engine can only read a portion of it. Orphan pages without any internal links pointing to them, as well as pages with too deep click depth, are the objects that crawlers such as GPTBot, ClaudeBot, and PerplexityBot are most likely to skip. They have a tighter crawl budget than Googlebot, return visits less frequently, and go shallower. No matter how good the content is, it will not be cited if it cannot be crawled.
Orphan pages and click depth, what are they talking about?
Orphan pages refer to pages within the website that cannot be reached by any internal link. It may exist in a sitemap, database or server. You can open it by directly entering the URL, but you can't get there no matter how you click from the homepage. The crawler mainly discovers pages by following links. If there is no path, it has no reason to visit.
Click depth is a related but different concept: it takes at least a few clicks to get from the home page to a certain page. The homepage itself is the zeroth layer, the pages directly connected to the homepage are the first layer, and so on. Pages with a click depth of more than three to four levels will be crawled significantly less frequently because crawlers prioritize limited resources for shallow pages with many inbound links.
Why AI crawlers are more ruthless about these two things
Traditional SEO also cares about orphan pages, but Googlebot has a huge infrastructure and is willing to crawl the website very deeply. It will also supplement the discovery path through various signals such as external links and browser data. This is not how most current AI crawlers work.
- Crawl budget is more conservative: most LLM crawlers only crawl a limited number of pages per visit, and deep pages are often not queued.
- Long interval between return visits: Googlebot may come back every day, and AI crawlers often return visits to the same site on a weekly or even monthly basis. If you miss one, you will have to wait a long time.
- Less reliance on external signals: they mainly follow the internal link structure of your website. If there is no internal path, there is almost no backup discovery mechanism.
- Limited rendering capabilities: Many AI crawlers do not execute JavaScript, so menus or links dynamically generated by JS cannot be seen by them, and those pages are essentially orphaned.
Three steps to run an internal link audit
The first step is to do a complete website crawl. Use Screaming Frog, Sitebulb or any crawler that can build a link graph to simulate crawler behavior starting from the homepage and capture the number of inbound internal links and click depth of each page. Focus on two columns: pages with zero inbound internal links are candidate orphans; pages with a click depth greater than three are crawling risk areas.
The second step is to compare the crawling results with the sitemap. Export all the URLs in sitemap.xml, then export the URLs that the crawler actually followed the link, and subtract the two lists. Pages that only appear in the sitemap but are not in the crawl results are almost certainly orphans. This step can uncover many missed pages that cannot be seen just by looking at the background.
The third step is to read the server logs. Filter out requests from user agents such as GPTBot, ClaudeBot, PerplexityBot, and Google-Extended, and compare which URLs they have actually captured. You will often find that the footprints of AI crawlers are concentrated on shallow pages, and the crawling records of deep or orphan pages are blank. The log does not lie, it directly tells you which content has never entered the AI's field of vision.
After finding the problem, how to fix it?
The direction of repair is not to fill every page with links, but to rebuild a reasonable discovery path so that important pages are closer to the homepage and cited by more related pages.
- Establish a topic hub (pillar) page: use one page to cover an overview of a certain topic, connect down to all subtopic pages, and gather scattered orphans into the same cluster.
- Add contextual in-text links: Naturally link to relevant in-depth pages in the text of the existing article. This kind of link with semantic context is particularly valuable for AI to determine the topic relevance.
- Flatten the website structure: Control the click depth of important pages within three levels, and adjust the directory structure or navigation menu if necessary.
- Check whether the navigation is generated by JavaScript: Use regular HTML links instead to ensure that crawlers that don't run JS can read it.
- Delete or merge truly useless pages: not every orphan is worth saving, so merge the ones that are worthless and leave the link weight to the pages that should be kept.
Prioritization is important. Start by working on pages that you want to be referenced by AI but are currently orphaned or deep, such as product comparisons, pricing descriptions, and in-depth guides. These are often the things that users will ask about in AI assistants and that you would most like to be quoted from.
When we audit for clients, the most common situation is that the best-quality in-depth content has never been caught by any AI crawler because it is only linked from a very deep directory. After pulling them into the topic hub, they will usually start to appear in the AI's answers within a month or two.— Tenten GEO Audit Observation
Treat internal links as search infrastructure
Orphan pages and click depth are not metaphysics, they are engineering problems that can be measured and fixed. Run a crawl, compare sitemaps, and read logs, and you will know how much content has never entered the AI's field of vision. If you want to quickly see which pages of your website have been missed by AI and how big the gap is, you can book a 30-minute GEO diagnosis. We will use actual crawling and log data to show you what content is being left out.
