Fixes for blocked bots

Pick the problem you see. Each fix has commands you can run on your own site.

• robots.txt returns errors, redirects or HTML: use this when crawling stops for the whole site, or bots crawl pages you blocked.
• Cloudflare blocks a crawler you meant to allow: use this when your site is on Cloudflare and a bot you want is blocked or challenged.
• 403 and 429 errors for Googlebot and Bingbot: use this when Search Console reports 403 errors on pages that load in a browser, or after a new firewall rule.