robots.txt AI Policy
A robots.txt AI policy is a set of User-agent directives in a website's robots.txt file that specifically addresses AI crawlers. This includes naming individual AI bots (GPTBot, ClaudeBot, PerplexityBot) with Allow or Disallow rules, rather than relying on wildcard rules designed for traditional search engine crawlers.
Want to see how your site performs against this benchmark?
Request a Demo