Self-hosted blog admin gives up on creating AI crawler identification guide
Understand this story
A self-hosted blog administrator, after analyzing site traffic, stated that about 8% is potential AI bot traffic (including impersonators), at roughly 20 requests per hour. Due to the large number and rapid changes of bots, he ultimately gave up on creating a unified identification guide, leaving only unfinished practical notes. As for alternatives, the administrator said basic rate limiting currently handles most of the blocking work and works well; Cloudflare Radar is a useful intelligence source but incomplete and unstable. He also said he does not pay for such services, so he cannot recommend paid sources.
AI overview based on source reports · Updated 7 hr ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIOverwhelmed by AI Bots
A self-hosted blog admin analyzing AI bot traffic found roughly 8% potential AI bot traffic (including impostors) at about 20 requests per hour, and gave up on building a simple admin guideline because the bots are too numerous and change too frequently. The notes cover three AI bot categories (crawler, assistant, search), Cloudflare Radar as the best but incomplete and fickle intel source, user-agent substring matching per RFC9309, an emerging JSON format for bot IP ranges, HTTP Message Signatures RFC9421, Petalbot's reverse-DNS approach, and JA4 as a weak signal. The author plans to try Fingerprint.js or Anubis later while relying on rate-limiting in the meantime.
Activity for this story
- Current comparable activity
- 8
- Peak comparable activity
- 10Nov 22 01:00
- Change in the past 24 hours
- –
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.