Hacker News AI· markosavran·· 11 hr agoAI score40
AI crawlers read my site more than Google does. A third of Googlebots were fake
AI crawlers read my site more than Google does. A third of Googlebots were fake
AI summary
A server-log analysis of just under two weeks found AI crawlers made almost three times as many requests as all search engines combined, and more than a third of "Googlebot" requests were fake, some from five rented Google Cloud servers using 30 bot names to probe for password and key files. AI crawlers don't run JavaScript, so analytics tools miss them, and no visitors arrived from ChatGPT, Perplexity, Claude or Gemini while Google sent 108. The author blocked no AI bots, only confirming files like .env return 403.
Source: Hacker News AI · opensource-analytics.com