How Do I Block AI Crawlers From My Site?

Blocking AI crawlers means adding disallow rules to a site's robots.txt file, naming each bot specifically. Common ones include GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, and Google-Extended, which covers Gemini and AI Overviews training separately from regular Googlebot.

Blocking these bots keeps a site out of that tool's training data or live citations, but it is an all-or-nothing switch per crawler. There is no way to allow citation while blocking training for the same bot on most platforms.

Most sites chasing AI visibility do the opposite: they check that these crawlers are explicitly allowed, since a leftover blanket disallow rule from years ago can quietly shut a site out of every AI answer engine.

Checking crawler access is one of the first things LLM optimization software verifies on a new site.