iCreatAI News
Skip to content
Trending storyWatching

Crawl Readiness releases first AI Access Index report

1 reports1 sources2 days agoUpdated

Understand this story

AI overview

Crawl Readiness releases the first AI Access Index (November 2026 edition, data as of November 1), examining the 684 responding sites among the top 1000 Tranco hostnames, checking their robots.txt declarations for 40 AI search crawlers and 9 training crawlers, as well as actual server responses. The report file is released under a CC BY-NC 4.0 license and may be shared and adapted free of charge for non-commercial purposes with attribution; commercial uses such as incorporating it into products or selling datasets require a license from [email protected]. Sites not in the sample can also be checked free of charge in the same way.

AI overview based on source reports · Updated 2 days ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 19
  1. Hacker News AI
    AI Access Index: how 684 of the top thousand hostnames treat AI crawlers

    The first edition of the AI Access Index, measured November 1, 2026, found that of 684 answering sites among the top 1,000 Tranco hostnames, 79.7% were open to AI crawlers, 8.8% allowed AI search but not training, 3.8% were selective and 7.7% were closed. Server behavior often diverged from robots.txt: 11.3% refused a crawler their robots.txt allows (8.2% excluding 429-only refusals), while 63.7% served a crawler their robots.txt blocks. Bytespider (17.5%) and CCBot (17.3%) were the most-blocked crawlers, and 15.8% of sites had a real llms.txt file.

Activity for this story

Current comparable activity
4
Peak comparable activity
10Nov 19 07:00
Change in the past 24 hours
-50%

Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.