Robots.txt Generator for AI Crawlers
Choose which AI crawlers may read your site and get a robots.txt to copy or download.
Set each AI crawler to allow, block or leave unnamed. The file updates as you choose. It runs in your browser. Nothing you enter is sent to Peach.
What this checks, and what it does not
It checks
- Builds a valid robots.txt from your choices, grouped so each crawler gets exactly one group.
- Repeats your closed paths inside every named group, because a crawler with its own group ignores the "*" group.
- Adds your sitemap line if you give one.
It does not
- It does not read your current robots.txt. Merge the output with rules you already have.
- It does not upload anything. You place the file at the root of your site.
- It cannot make a crawler obey.
How to read the result
A crawler left as "Not named" gets no group of its own and follows the "*" rules. That is the right choice for most crawlers on most sites.
Blocking a training crawler and allowing a search crawler from the same company is a common setup: the first decides whether your pages are used to build models, the second whether they can be found and cited in an answer.
More in the docs: Site audit in the Peach docs.
Related tools
- AI Crawler Access Checker (Live, input: web address): Reads a site's robots.txt and reports what it says to 16 AI crawlers, with the rule behind each result.
- llms.txt Generator (Live, input: short form): Fill in a name, a summary and your key pages. Get an llms.txt in the proposed format.
- Sitemap Checker (Live, input: web address): Finds a site's sitemap through robots.txt or the default address and reports what is in it.
Frequently asked questions
Where does the file go?
At the root of the host, so it is reachable at yourdomain.com/robots.txt. A file in a subfolder is not read.
Why are my closed paths repeated in the AI crawler group?
A crawler obeys only the most specific group that names it. If GPTBot has its own group, it never reads the "*" group, so the paths have to be listed again.
How do I check the result?
Publish the file, then run the AI Crawler Access Checker on your domain. It reads the live file and shows which rule applies to each crawler.
Check whether AI names your brand
Peach writes the questions a buyer would ask about your category, sends them to AI engines, and shows every answer and who it names. Run the AI visibility check. It needs a free account.