Let Perplexity catch you: robots, llms.txt and Perplexity crawler settings

There are only two prerequisites for Perplexity to cite you: its crawler can capture your page, and it can understand what you are talking about. Most people get stuck at the first level without realizing that a line of robots.txt rules with wildcards, CDN's default protection, or a user agent that is not allowed is enough to completely make you disappear from Perplexity's answers. Releasing PerplexityBot is the minimum threshold, not a guarantee of being cited; but without it, all subsequent GEO work will be empty talk.

Let’s figure it out first: Perplexity sends two crawlers with completely different behaviors.

Many people think that Perplexity only has one crawler and only blocks one name when setting it, so they think it is closed or opened, but the result is completely wrong. Perplexity official documents list two agents: PerplexityBot is responsible for building search indexes and determining whether your page is eligible to appear in the results. It will comply with robots.txt; Perplexity-User is sent when the user asks a question and the system immediately crawls a specific URL. Because it responds to the user's direct action, it is usually not subject to robots.txt. This difference is critical: if you block crawlers in robots.txt, you can block indexing, but you may not be able to block instant crawling.

• PerplexityBot: Used to build and update Perplexity's search index, determine whether you can be listed as a candidate source, and comply with robots.txt rules.
• Perplexity-User: Instantly visit a specific page when the user asks a question. This is an action in response to the user. Generally, robots.txt blocking is not applied.
• Verify authenticity: Both may be impersonated. Don’t just look at the user agent string. You should confirm it with the IP segment published by Perplexity or reverse DNS (resolved back to perplexity.ai).
• Common misunderstanding: being released in robots.txt does not mean that you will be caught or cited, it only makes you eligible to enter the candidate pool.

robots.txt: The place most commonly accidentally killed by one-line settings

When we do GEO audits for clients, the most common first fatal flaw is robots.txt. Some sites use User-agent: * combined with Disallow: / to block the whole site and forget to open it after going online; some block main content directories such as /blog/ or /resources/; more hidden are the Bot management or firewall rules of platforms such as Cloudflare and Vercel. If you do not change robots.txt, they will directly return 403 to PerplexityBot at the edge layer. This type of question will not prompt you with an error message, it will just make you silently absent from Perplexity’s answers. The minimum feasible correct setting is actually very short: explicitly allow PerplexityBot in the robots.txt in the root directory, and point to your sitemap.

• Clear permission: Write a paragraph User-agent: PerplexityBot followed by Allow: / to avoid being affected by the blocking rules of User-agent: *.
• Specify sitemap: Add Sitemap: https://yourdomain/sitemap.xml at the end of the file to let the crawler get the complete URL list at once.
• Check CDN and WAF: Confirm that Cloudflare, Vercel, and AWS WAF do not block all known AI crawlers at the edge layer.
• Don’t bother with noindex and want to be cited: If a page has a noindex in the robots meta, it is equivalent to saying “Don’t include me” to search engines and Perplexity at the same time.
• Verify with logs: Go to the server access log and search for PerplexityBot to confirm that it has actually come and got 200 instead of 403 or 429.

llms.txt: It’s worth spending ten minutes doing, but don’t think of it as an engine entry point

llms.txt is a proposal proposed by Jeremy Howard of Answer.AI in 2024: Put a Markdown file of /llms.txt in the root directory of the website, and list your most important pages and descriptions in a streamlined structure to facilitate large language models to quickly understand the key points of the site. The intention is good, but to be honest - as of now, no mainstream engine including Perplexity and Google has publicly announced that it will actually consume llms.txt. Google's John Mueller also publicly stated that no search engine uses it as a basis for crawling or ranking. Should you still do it? My advice is do it, but be honest about its role. It is extremely low-cost, and one file can list the pillar content, product descriptions and contact information; it is an additional layer of clean entry for agents that will read it or engines that may be supported in the future; the more practical value is that writing llms.txt will force you to think clearly about "which pages are the core", which itself is helpful to GEO. Don't put all your budget and expectations on it. What really determines whether Perplexity can catch you or not is robots.txt and clean HTML.

What you can catch is just the ticket, the point is to let Perplexity "choose you as the source"

Perplexity When answering a question, it is common to cite only a few sources. It prefers passages that directly correspond to the question and are easy to extract cleanly. If your answer is buried in the eighth paragraph, with a lot of "with the rise of AI" in front of it, or if one paragraph talks about three things at the same time, it will be difficult for the engine to cut you out and use it as a quote. The conclusion comes first, one paragraph tells one thing, and the title is the answer to the question. These writing methods are not insisted by young people, but they directly affect whether you are selected.

• Answer front: Give the conclusion in the first sentence of each section, and put the why and how to do it at the end.
• The title is the question: use H2 and H3 to write sentences that users will actually ask, so that the engine can match the query.
• Paragraphs are self-sufficient: one paragraph only talks about one key point, and does not assume that the reader has read the previous paragraph, so it can be quoted separately.
• Upload structured data: Articles are linked to Article, and FAQPage schema is used in the Q&A area to help the engine confirm that the content type corresponds to the Q&A.
• Give numbers and context: Verifiable facts are easier to quote than adjectives.

Freshness: Perplexity Specially Eats "Latest Update"

Perplexity positions itself as a tool for answering current, time-sensitive topics, with a clear preference for recent content. This means two things: first, the page must truthfully provide dateModified, and do not let an article updated last year appear to be three years ago; second, the core page must have a real update rhythm, adding new numbers and new practices, rather than changing the date to deceive the engine. When we maintain content engines for customers, we will regularly visit high-value pages to supplement the latest execution experience, which makes a practical difference in Perplexity's source selection.

Whether it can be cited by Perplexity is 90% determined by technical aspects: it can be grasped, readable, and new enough. No matter how good the content is, there is no chance that crawlers will be able to get in.— Tenten GEO Consulting Team

30-minute self-check before going live

1. Open "your domain/robots.txt" in the browser and confirm that the entire Disallow: / section is not blocked by mistake, and that PerplexityBot is clearly allowed.
2. Search the server logs for PerplexityBot, confirm that there has been a recent visit and return 200.
3. Check the Bot rules of CDN and WAF to confirm that AI crawlers are not blocked at the edge layer.
4. Randomly check the three main articles to see if the conclusion comes first in the first paragraph and whether the title is appropriate to the real issue.
5. Confirm that the sitemap can be accessed normally, dateModified is correct, and put llms.txt as the content map.

Most of the items on this list can be fixed in an afternoon. The difficult thing is to know which items you have missed, especially those problems hidden in CDN rules and platform default blocks. You will not be able to see them unless you take the initiative to check. If you want to confirm whether Perplexity can catch you now and what opportunities it has missed to be cited, you can go to /contact to make an appointment for a 30-minute GEO diagnosis. We will directly look at the robots, logs and content structure of your website, and list the gaps for you one by one.