Study: Websites block AI training crawlers far more than answer crawlers
Understand this story
Trakis Research analyzed 13,938 websites with readable robots.txt files and found that the proportion of websites blocking AI training crawlers is about three times that of blocking answer crawlers. By crawler name, among sites blocking GPTBot, only 17.1% also block the ChatGPT search crawler OAI-SearchBot, and 25.9% block ChatGPT-User. In other words, most websites that block training crawlers do not also block ChatGPT's answer or search crawling. The study data is released under CC BY 4.0, and the code under the MIT license; websites wishing to be removed from the data can email [email protected].
AI overview based on source reports · Updated 2 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIBlock training, keep answers? What 13,938 sites ask of AI bots
Trakis Research analyzed robots.txt files from 13,938 readable sites in Common Crawl's September 2026 archive and found that of sites blocking OpenAI's GPTBot by name, 17.1% also block OAI-SearchBot and 25.9% block ChatGPT-User. The figure splits sharply by method: sites using Cloudflare's managed robots.txt rarely block answer bots (0.3% and 1.0%), while sites writing their own rules block them far more often (31.3% and 47.0%). Training bots are blocked about three times as often as answer bots, while Googlebot and Bingbot are almost never blocked.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 10Nov 4 22:00
- Change in the past 24 hours
- -49%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.