Webmaster Log Statistics: AI Crawler Requests Far Exceed Search Engines
Understand this story
Webmaster markosavran analyzed nearly two weeks of nginx logs and found that AI crawler request volume was nearly 3 times the combined total of all search engines; more than one-third of requests claiming to be Googlebot were fake. These fake crawlers mostly came from 5 rented Google Cloud servers, using 30 crawler names in total to send 2,496 requests, first reading the homepage, robots.txt, and sitemap, then probing files where passwords and keys are stored, but none leaked. The identification method he shared is: search the user agent in the server access logs for AI crawler names such as GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and meta-externalagent. He said crawlers that collect text to train new models can be refused in robots.txt; fake crawlers ignore robots.txt, so it is necessary to ensure the server denies files such as .env. He said he did not block any AI crawlers, and only checked that files such as .env return 403.
AI overview based on source reports · Updated 11 hr ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIAI crawlers read my site more than Google does. A third of Googlebots were fake
A server-log analysis of just under two weeks found AI crawlers made almost three times as many requests as all search engines combined, and more than a third of "Googlebot" requests were fake, some from five rented Google Cloud servers using 30 bot names to probe for password and key files. AI crawlers don't run JavaScript, so analytics tools miss them, and no visitors arrived from ChatGPT, Perplexity, Claude or Gemini while Google sent 108. The author blocked no AI bots, only confirming files like .env return 403.
Activity for this story
- Current comparable activity
- 7
- Peak comparable activity
- 10Nov 17 23:00
- Change in the past 24 hours
- –
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.