AI Crawler Access Checker
Reads a site's robots.txt and reports what it says to 16 AI crawlers, with the rule behind each result.
Enter a site. Peach fetches its robots.txt and reports, crawler by crawler, whether the file asks it to stay away. It makes one request for a public page. It needs JavaScript to run; the explanation below does not.
What this checks, and what it does not
It checks
- The robots.txt file at the root of the site, fetched once.
- Sixteen AI crawlers, each matched to the group that names it, or to the "*" group if none does.
- Whether the homepage is disallowed, whether only some paths are, and the line that decides it.
It does not
- It does not test a firewall, CDN or bot-protection rule. A site can allow a crawler in robots.txt and still refuse it at the network edge.
- It does not prove a crawler obeys the file. robots.txt is a request.
- It does not say whether an AI engine names your brand. A crawler being allowed is a precondition, not a result.
How to read the result
"Allowed" means nothing in the crawler's group disallows anything. "Some paths closed" means the homepage is open and specific paths are not. "Whole site closed" means the rule that applies to "/" is a Disallow.
Crawlers are grouped by what their operator says they are for. A training crawler collects pages to build a model. A search crawler builds the index an AI engine draws on when it answers. A user-requested fetch happens when someone pastes your address into a chat. Closing the first does not close the other two.
If the site refuses Peach's request for the file, no verdict is given. A crawler the site recognises may be sent a different file.
More in the docs: Site audit in the Peach docs.
Related tools
- Robots.txt Generator for AI Crawlers (Live, input: short form): Choose which AI crawlers may read your site and get a robots.txt to copy or download.
- Can AI Read My Page? (Live, input: web address): Fetches a page without running JavaScript and counts the words a non-rendering crawler receives.
- Sitemap Checker (Live, input: web address): Finds a site's sitemap through robots.txt or the default address and reports what is in it.
Frequently asked questions
What happens if a site has no robots.txt?
Nothing is disallowed. Crawlers treat a missing file as permission to fetch any page, so every crawler is reported as allowed.
Does blocking GPTBot remove my site from ChatGPT answers?
Not by itself. GPTBot collects pages for training. ChatGPT search uses a separate crawler, OAI-SearchBot, and a page a user asks about is fetched by ChatGPT-User. The three are listed separately for that reason.
Is Google-Extended a crawler?
No. It is a control token that Googlebot reads. Disallowing it opts a site out of Gemini training and does not change Google Search.
Check whether AI names your brand
Peach writes the questions a buyer would ask about your category, sends them to AI engines, and shows every answer and who it names. Run the AI visibility check. It needs a free account.