https://crawlguide.com/ Find out which crawlers can reach your pages, and fix what blocks them. en-us https://crawlguide.com/fixes/403-429-verified-crawlers/ https://crawlguide.com/fixes/403-429-verified-crawlers/ Thu, 22 Oct 2026 00:00:00 GMT Google drops pages that return 403, and slows crawling of the whole host when many URLs return 429. Let verified crawlers past the rule that sends them. https://crawlguide.com/fixes/cloudflare-ai-bot-settings/ https://crawlguide.com/fixes/cloudflare-ai-bot-settings/ Thu, 22 Oct 2026 00:00:00 GMT Several Cloudflare bot settings can block crawlers. Setting Training to Block now stops Googlebot and Bingbot too. Find the one that blocked your crawler. https://crawlguide.com/fixes/robots-txt-status-codes/ https://crawlguide.com/fixes/robots-txt-status-codes/ Thu, 22 Oct 2026 00:00:00 GMT Serve robots.txt as a plain-text file with status 200. A 4xx can mean no rules, a 5xx can stop crawling, and a web page instead usually means no rules. https://crawlguide.com/methodology/ https://crawlguide.com/methodology/ Thu, 22 Oct 2026 00:00:00 GMT The rules our robots.txt checker and tester follow, where they come from (RFC 9309 and Google), and what a robots.txt check cannot tell you.