Corrections policy
Effective
When we get something wrong, we fix it and say so.
- On a page, we fix the text and add “Update:” at the top, with the date, what changed and what it said before.
- Typo fixes do not get an Update line.
- In the crawler list, each entry shows when we last checked it, and the list shows when we last re-checked the source quotes.
- When an operator’s documentation changes, we update the entry and its date.
- In the tester, we fix wrong verdicts, add the case to our tests and log it in the method.
To report an error, use the contact page.
Editorial standards
CrawlGuide covers crawler access for search and AI: which crawlers exist, how robots.txt applies to them, and the server and CDN settings that block them. We do not compare platforms or give content advice.
Facts come from the crawler operators’ and CDN providers’ own documentation, from standards such as RFC 9309, and from tests we run ourselves. Articles, the method and the crawler list give their sources with the date we read them. A test says how and when it ran, and the scripts are published with the page.
No vendor pays us or gives us free access beyond what it offers everyone. The disclosure page explains how we use Cloudflare, which hosts the site, registers its domain and forwards its mail.
The publisher, Jonathan Greenfield, is responsible for what CrawlGuide publishes. The About page has his details.