Study: About one in five HTTP 200 responses contain no usable content
Understand this story
A new study estimates that about 20.9% of web responses returning HTTP 200 actually contain no usable body text, instead being empty-shell pages, CAPTCHA checks, access-restriction pages, or substitute pages. Over three days, the study tested 10,307 URLs with a basic crawler and a browser capable of executing JavaScript; empty-shell pages and error/substitute pages together accounted for more than two-thirds of failure cases. This means crawlers cannot reliably judge a page as valid based solely on status code 200, and AI web scraping needs extra verification that content is actually usable.
AI overview based on source reports · Updated 2 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AITwenty Percent of all 'ok' web responses AI gets are not okay
A new paper estimates that 20.9% of pages returning HTTP status 200 to a static client contain none of the page's main content, instead delivering app shells, bot challenges, block pages, error or placeholder pages, and partial content. Testing 10,307 web addresses over three days, the study found apparent citation drift rose from 5.8% to 21.9% when all 200 responses were accepted as successful retrievals, and recommends that fetching tools report what was actually retrieved rather than relying on status codes.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 10Nov 4 23:00
- Change in the past 24 hours
- -50%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.