Hacker News AI· floxolab·· 1 days agoAI score34
Show HN: I built a tool to check website access for 14 AI crawlers
Show HN: I built a tool to check website access for 14 AI crawlers
AI summary
FloxoLab's AI Crawler Access Checker takes a URL, reads its robots.txt under RFC 9309, requests the page with the user agents of 14 AI crawlers, and compares each response with a normal browser request to give a per-crawler verdict. It also reports page directives, text available without JavaScript, and discovery files such as /llms.txt and the sitemap, while noting limits like vendor IP verification and bot challenges returned with HTTP 200. The tool is free with no account and allows 8 checks per 10 minutes.
Source: Hacker News AI · floxolab.com