About CrawlGuide
Published
CrawlGuide helps you check if bots such as Google and ChatGPT can read your website. It covers search and AI bots, including the ones AI assistants send when a person asks about a page.
What we publish
- A robots.txt checker, where you type your website, and a robots.txt tester, where you paste your file. Both follow RFC 9309, the robots.txt standard.
- A crawler list built from each operator’s own documentation, with the date we last checked it.
- Fixes for blocked bots: CDN bot settings, firewall rules, error responses and redirects.
We do not compare platforms or give content advice.
How we check our work
When we test something ourselves, the page says so and gives the date and the commands. When a fact comes from a company’s documentation, the page names it. When we do not know, we write “unknown”. The method explains how the tester decides, and the corrections policy explains how we fix mistakes.
Who publishes CrawlGuide
CrawlGuide is published by Jonathan Greenfield, who is responsible for what it publishes. He also publishes other sites, most of them about search and AI.
- Email: hello@crawlguide.com (the contact page says what to include)
- Postal address: 71-75 Shelton Street, London WC2H 9JQ, United Kingdom
Money and independence
CrawlGuide is independent of the crawler operators and CDN providers it covers. None of them owns it or pays for what it says, and it carries no ads or affiliate links. The site is hosted on Cloudflare, and its domain is registered there at the standard price, as the disclosure explains.