Crawl Readiness releases first AI Access Index report
Understand this story
Crawl Readiness releases the first AI Access Index (November 2026 edition, data as of November 1), examining the 684 responding sites among the top 1000 Tranco hostnames, checking their robots.txt declarations for 40 AI search crawlers and 9 training crawlers, as well as actual server responses. The report file is released under a CC BY-NC 4.0 license and may be shared and adapted free of charge for non-commercial purposes with attribution; commercial uses such as incorporating it into products or selling datasets require a license from [email protected]. Sites not in the sample can also be checked free of charge in the same way.
AI overview based on source reports · Updated 2 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIAI Access Index: how 684 of the top thousand hostnames treat AI crawlers
The first edition of the AI Access Index, measured November 1, 2026, found that of 684 answering sites among the top 1,000 Tranco hostnames, 79.7% were open to AI crawlers, 8.8% allowed AI search but not training, 3.8% were selective and 7.7% were closed. Server behavior often diverged from robots.txt: 11.3% refused a crawler their robots.txt allows (8.2% excluding 429-only refusals), while 63.7% served a crawler their robots.txt blocks. Bytespider (17.5%) and CCBot (17.3%) were the most-blocked crawlers, and 15.8% of sites had a real llms.txt file.
Activity for this story
- Current comparable activity
- 4
- Peak comparable activity
- 10Nov 19 07:00
- Change in the past 24 hours
- -50%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.