iCreatAI News
Skip to content
Hacker News AI· grubbomatic·· 2 days agoAI score58

AI Access Index: how 684 of the top thousand hostnames treat AI crawlers

AI Access Index: how 684 of the top thousand hostnames treat AI crawlers

AI summary

The first edition of the AI Access Index, measured November 1, 2026, found that of 684 answering sites among the top 1,000 Tranco hostnames, 79.7% were open to AI crawlers, 8.8% allowed AI search but not training, 3.8% were selective and 7.7% were closed. Server behavior often diverged from robots.txt: 11.3% refused a crawler their robots.txt allows (8.2% excluding 429-only refusals), while 63.7% served a crawler their robots.txt blocks. Bytespider (17.5%) and CCBot (17.3%) were the most-blocked crawlers, and 15.8% of sites had a real llms.txt file.

Source: Hacker News AI · crawlreadiness.com