Hiding your site from AI crawlers

Hide your site from artificial intelligence crawlers while leaving the site available for visitors to access.

You can hide your site from AI web crawlers, while leaving it available for visitors to access.

To hide your site from search engine crawlers, visit Hiding your site from search engine results. To review all options for hiding content on your site, visit Controlling who can access your site's pages.

What are crawlers?

AI companies use web crawlers to train their large language models (LLMs). These models learn from the content of websites they scan, and use the data to improve the accuracy and general capabilities of their AI models.

Hiding your site from search engines or AI crawlers

By default, AI crawlers scan all content published on the Internet, but if you'd like to exclude your website from these scans, you can choose to hide your site from certain AI crawlers. To hide your site:

1. Open the Settings panel.
2. Click Crawlers.
3. Switch the Artificial Intelligence crawlers toggle off.

Switching the Artificial Intelligence crawlers toggle off updates your robots.txt file to block the following bots from crawling your site:

• Anthropic AI
• CCBot
• Google Extended
• GPTBot and ChatGPT-User

Note: It's currently not possible to hide specific pages from AI crawlers. Hiding your site from AI crawlers doesn't retroactively remove content previously scraped from a site from AI model training data.