Block training, keep answers? What 13,938 sites ask of AI bots

Trakis Research analyzed robots.txt files from 13,938 readable sites in Common Crawl's September 2026 archive and found that of sites blocking OpenAI's GPTBot by name, 17.1% also block OAI-SearchBot and 25.9% block ChatGPT-User. The figure splits sharply by method: sites using Cloudflare's managed robots.txt rarely block answer bots (0.3% and 1.0%), while sites writing their own rules block them far more often (31.3% and 47.0%). Training bots are blocked about three times as often as answer bots, while Googlebot and Bingbot are almost never blocked.