Site Audit and AI crawlers

Whether the crawlers behind ChatGPT, Gemini, Claude and Perplexity are allowed to read your site, and what your homepage tells them.

Updated 2026-10-06

In short. Site Audit reads your robots.txt and reports whether five AI crawlers are allowed in. It also shows what a crawler can extract from your homepage. A blocked crawler cannot learn about you from your own site.

The crawlers Peach checks

Crawler name in robots.txt	Who runs it
`GPTBot`	OpenAI, for ChatGPT
`Google-Extended`	Google, for Gemini and AI features
`ClaudeBot`	Anthropic, for Claude
`anthropic-ai`	Anthropic, an older name
`PerplexityBot`	Perplexity

For each one, the audit says allowed or blocked. When a crawler is blocked, it shows the rule in your robots.txt that blocks it, so you know which line to change.

How to read the result

All allowed. Nothing to do here.
One or more blocked. Decide whether that is deliberate. Some sites block AI crawlers on purpose. If you want AI answers to draw on your site, a block works against you.
No robots.txt. Crawlers treat a missing file as permission to read everything.

A rule under User-agent: * applies to every crawler that has no group of its own, including these.

Allowing a crawler

To allow a crawler explicitly, give it its own group:

User-agent: GPTBot
Allow: /

Repeat the group for each crawler you want to allow.

What AI understands about your site

The audit also shows the information an AI crawler can extract from your homepage. If your homepage shows almost nothing without JavaScript, a crawler sees almost nothing too.

This matters twice over. It affects what crawlers learn, and it affects Peach: the classification behind your prompts and competitors starts from the same page.

What the audit does not do

It checks permission, not behaviour. Being allowed in does not mean a crawler has visited.
It reads robots.txt only. It does not check noindex tags, login walls or firewalls.
Allowing crawlers does not raise your mention rate by itself. It removes a barrier. What gets you named is content that answers the question.

All docs pages

Getting started

What Peach is: Peach checks whether AI answers name your brand when buyers ask about your category, and helps you write the content that is missing.

Your first free check: Run Peach on your website. It needs a free account and no credit card, and takes about a minute.

Reading your report: What each tab of a Peach report shows, and how to read a verdict without over-reading it.

AI Visibility

How the score works: What Peach samples, how a mention is detected, and exactly how the mention rate is calculated.

Engines and what each one runs: Exactly what Peach asks ChatGPT, Gemini and Google, and how that differs from what you see in their apps.

Missed questions: A missed question is a buyer question that got an answer from at least one engine, where no engine named you.

Competitors and how they're chosen: How Peach decides who you compete with, how to correct the list, and what the citation rate beside each name means.

Citations: What the Citations tab counts, and why a citation in Peach is a domain written in an answer, not a source link.

Prompts: The buyer questions Peach asks, how they are written, and how to add your own.

Query Fan-Out: An estimated set of related questions behind each prompt. It is generated, not observed, and Peach labels it that way.

Site Audit and AI crawlers: Whether the crawlers behind ChatGPT, Gemini, Claude and Perplexity are allowed to read your site, and what your homepage tells them.

Market and geography: Which country your report is measured in, how Peach decides, and how to set it yourself in Brand Hub.

Content Engine

Content Engine overview: The workspace where a missed question becomes a topic, an outline and an article. What the Summary tab shows and how sources work.

What to write: How Peach turns your missed questions into ranked article suggestions, and what High, Medium, Low, Open and stale mean.

New article: Start an article from a blank title. Which formats are available, which are coming soon, and what top of funnel and mid funnel change.

The outline, the article and quality checks: Edit the outline before anything is written, then review the article against checks that run on every save.

Publishing

Publishing connectors: Where a finished article can go, and the real state of each connector today.

What "Publish again" does: No connector updates a post that is already published. What publishing a second time does on each destination, before you click.

Brand Hub

Brand Hub: Where you tell Peach what you sell, who you compete with, how you sound and where you compete. Everything else reads from it.

Account

Plans, quotas and billing: What a free account includes, the two paid plans, how the article quota is counted and how billing works.

Reference

Glossary: Plain definitions of the terms Peach uses, including the difference between a mention and a citation.

Limitations and accuracy: What a Peach report can and cannot tell you. A sample is not a census, and an API answer is not the app.

FAQ: Short answers to the questions people ask most about Peach checks, scores, articles and billing.

Changelog: What has shipped in Peach, newest first, with the date each change was made.