Developer releases AI crawler access check tool
Understand this story
Developer floxolab released AI Crawler Access Checker: after entering a URL, the tool reads that site's robots.txt and requests the page using the user agents of 14 AI crawlers, then compares the results with a normal browser request to give a conclusion for each crawler as allowed, partially restricted, or blocked. The tool is free, requires no account, and is limited to 8 checks every 10 minutes; it only checks ports 80 and 443, rejects private, local, and intranet addresses, reads no more than 3 MB of raw HTML, and does not follow pages that require login. The developer says checks are performed per request, reports are returned only to the user's browser, and no URLs or results database are stored.
AI overview based on source reports · Updated 1 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIShow HN: I built a tool to check website access for 14 AI crawlers
FloxoLab's AI Crawler Access Checker takes a URL, reads its robots.txt under RFC 9309, requests the page with the user agents of 14 AI crawlers, and compares each response with a normal browser request to give a per-crawler verdict. It also reports page directives, text available without JavaScript, and discovery files such as /llms.txt and the sitemap, while noting limits like vendor IP verification and bot challenges returned with HTTP 200. The tool is free with no account and allows 8 checks per 10 minutes.
Activity for this story
- Current comparable activity
- 4
- Peak comparable activity
- 10Nov 13 14:00
- Change in the past 24 hours
- -50%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.