Training vs Retrieval Crawler

AI crawlers fall into two fundamental categories: training crawlers that ingest content into model weights (GPTBot, Google-Extended, CCBot) and retrieval crawlers that fetch content for real-time AI answers (ChatGPT-User, PerplexityBot, Claude-SearchBot). The distinction matters because blocking a training crawler prevents your content from entering future models, while blocking a retrieval crawler prevents your content from appearing in real-time AI answers.