Privacy notice

The personal data we handle is:

• the technical data your browser sends when you visit a page
• the emails people send us
• our research data, which can identify people

Who is responsible for your data

CrawlGuide is published by Jonathan Greenfield, who is responsible for the personal data described here. Data protection law calls this person the “controller”.

• Email: hello@crawlguide.com
• Postal address: 71-75 Shelton Street, London WC2H 9JQ, United Kingdom

What we collect and why

Data	Why we use it	Legal basis	How long we keep it
Request data: IP address, page, time, browser and device	To deliver pages and protect the site	Legitimate interests	31 days in our dashboard
Your email: address, name, message and attachments	To reply, and to correct a page you report	Legitimate interests	2 years after the last message
Research data: AI answers and what the services show with them, such as the pages they cite, search results, posts, business listings and reviews. Any of these can name a person.	To publish research on which sources AI answers use	Legitimate interests	See “How long we keep research data”

When you visit the site

Your browser sends technical data with every request to the site, such as for a page or an image. This includes the page it wants and your IP address, the number that identifies your internet connection.

Cloudflare hosts the site. It uses this data to deliver pages and to block attacks. If it suspects an attack, it may ask you to pass a security check. It may then store a cookie, a small file on your device, to record that you passed.

Cloudflare’s dashboard shows us traffic totals. For a sample of requests, it also shows details such as IP addresses. We can see these for up to 31 days.

We use this data only to keep the site working and secure. We do not try to identify visitors.

No advertisers collect data here. To protect its network, Cloudflare uses request data, such as IP addresses, from the many sites it serves. We do not track you across websites. So if your browser sends a Do Not Track signal, it changes nothing here.

When you email us

Email to hello@crawlguide.com passes through Cloudflare’s Email Routing to our inbox at Google. Cloudflare does not store the message. Its logs keep the sender, recipient and subject for 31 days.

We use your email to reply to you. If you report a mistake, we also use it to correct the page. We never add you to a mailing list.

We delete emails 2 years after the last message. We keep them longer only while a complaint or legal claim about them is still open.

Our research data

We publish research on which web pages AI services cite when they answer our questions. ChatGPT and Google’s AI answers are examples of these services.

How we collect it

A data collection service asks the AI services our questions for us. We may also check an answer by hand.

We save the text of each answer and what the service shows with it. That includes the address, title and summary of each page it cites or lists. It also includes any search results, videos, posts, business listings and reviews shown beside the answer. We may keep a copy of the answer page for up to 30 days, so the answer can be checked.

The AI services are open to the public, and the pages and listings they show were publicly available. Some of this data identifies a person, for example:

• a social media profile
• a personal blog
• a forum user’s page
• a name in an answer
• a review someone wrote
• a sole trader’s business listing, with its phone number

What we publish from our research

Unless a limit below applies, we publish the full address of:

• pages from organizations and businesses
• articles with a byline on a publication
• blogs where people write about their work
• discussion threads about a product or topic

For a personal profile or other personal page, we publish only the website and the type of page, such as “linkedin.com, personal profile”. Sometimes we publish only the type of page.

We never publish:

• anyone’s contact details
• information about anyone’s health, beliefs, sex life or any crime
• what an answer says about a private individual

Our full records stay in our private files.

How long we keep research data

• Published data: while the report it’s in is online.
• Full records: while a published report relies on them. We review them once a year.
• Records that no report has used: we delete them 12 months after we collected them.
• Copies of answer pages: up to 30 days.

If your page or name is in our research

We cannot tell each person whose page or name appears in an answer. We usually have no way to reach them, and finding their contact details would mean collecting more data about them. This notice is how we tell them.

If your page or name is in our published data, email hello@crawlguide.com. We will remove it within 1 month, and leave it out of later reports. You can also object to our use of it (see “Your right to object”).

The robots.txt checker and tester

When you check a website

When you type a website’s address on the home page, our server fetches that site’s public robots.txt file once. It uses the file only to show you the result, and does not save it.

The robots.txt tester does the same when its page address ends in ?site= and a site name. The home page’s check sends you there when a file is too long for a link.

The address you typed is in the page’s link and in the request your browser sends to our server. So it can appear in the request data Cloudflare shows us for up to 31 days.

To limit how often one visitor can run checks, our server counts the requests from each IP address. It keeps the counts in its memory only:

• for an IPv6 address, it counts only the first half
• it starts the counts again each minute, and drops older counts at the next request
• it never writes the counts anywhere

When you paste a robots.txt file

When you paste a robots.txt file into the tester, the test runs in your browser. The file, page address and bot names you enter are not sent to us or anyone else.

After a test runs, the page’s address holds what you entered, after the #, unless the file is too long to fit. Browsers do not send that part to servers. But anyone you share the address with can read it.

Our legal basis

The law requires a legal reason, called a “legal basis”, for using personal data. We rely on legitimate interests (UK GDPR Article 6(1)(f)). Our interests are:

• keeping the site working and secure
• answering what people send us
• publishing research on which sources AI answers cite

Readers can use our research to see which sources AI answers cite. We use only the data these interests need, in ways readers would expect. For the research, we publish only what it needs. We have judged that your interests do not override ours.

No law or contract requires you to give us personal data. But your browser must send its IP address to receive pages, and we need your email address to reply.

We never let a computer make a significant decision about you.

Who else handles your data

Cloudflare, Inc. hosts the site and forwards email. On our instructions, it handles the pages, your email and the logs it shows us. It decides for itself how to use the rest of the request data, for example to protect its network. It does this under Cloudflare’s privacy policy.

Google LLC provides the inbox where your email is stored. Google handles it under its own privacy policy.

For our research data, a data collection service puts our questions to the AI services and collects their answers. It does this for us, on our instructions.

GitHub, a service that stores files and runs automated jobs, keeps our research records in a private repository in our account. This includes the results our automated jobs on GitHub save there. GitHub handles them under its own privacy statement.

We analyze the records with Claude, Anthropic’s AI assistant. Claude processes them on Anthropic’s servers. Anthropic handles what we give Claude under its own privacy policy.

We do not sell personal data or use it for advertising. Apart from the research data we publish, we pass it to no one else unless the law requires it.

When your data leaves the UK

UK law limits when personal data can leave the UK. It treats some countries and schemes as “adequate”, which means it accepts that they protect data well enough. Other transfers usually need a contract in a form UK law approves, such as standard contractual clauses with the UK’s addendum.

Cloudflare and Google are US companies and may process your data in the US and other countries.

Cloudflare relies on its certification under the UK Extension to the EU-US Data Privacy Framework. UK law treats US companies with this certification as adequate. Cloudflare’s fallback is standard contractual clauses.

Google describes its safeguards, including standard contractual clauses, on its data transfer frameworks page.

GitHub is a US company and may store our research records in the US and other countries. It is certified under the UK Extension to the EU-US Data Privacy Framework. UK law treats US companies with this certification as adequate.

Our data collection service is a company in Malta. UK law treats Malta as adequate. Its data processing terms cover any transfer of our research data out of the UK. They use standard contractual clauses, with the UK’s addendum to them.

Anthropic stores what we give Claude on its servers in the US and other countries. Its privacy policy says it relies on adequacy decisions and standard contractual clauses, including their UK equivalent, when it moves personal data abroad.

Ask us for more detail.

Your rights

You can ask us:

• for a copy of the personal data we hold about you
• to correct it
• to delete it
• to restrict how we use it

Write to hello@crawlguide.com or to the postal address above. It’s free.

We reply within 1 month of your request. If we need something to confirm who you are, the month starts when we receive it. A complex request can take up to 2 more months, and we will tell you why.

Your right to object

We rely on legitimate interests, so you can object to our use of your personal data at any time. Email hello@crawlguide.com and we will stop, with 2 exceptions. We may continue if we have compelling legitimate reasons that override your interests and rights. We may also continue if we need the data for a legal claim.

How to complain

If you’re unhappy with how we’ve handled your personal data, you can complain to us by email or by mail. Use the details above. We will acknowledge your complaint within 30 days, and tell you the outcome without undue delay.

You can also complain to the UK regulator, the Information Commission, known as the ICO. You can complain online at ico.org.uk/make-a-complaint or call 0303 123 1113.

Changes to this notice

When the way we handle personal data changes, we update this notice. The date of the change shows at the top of the page. If a change affects data we already hold, we email the people concerned first. If we do not have their email address, we show the change here.