iCreatAI News
Skip to content
Trending storyUpdating

Chris Shiflett: AI crawlers scrape HTML, OpenAI partially runs scripts

1 reports1 sources10 hr agoUpdated

Understand this story

AI overview

Chris Shiflett points out that AI crawlers such as GPTBot and ClaudeBot are similar to Googlebot in 2004: they mainly scrape HTML and largely do not execute JavaScript. He says OpenAI's crawler recently began running scripts on some websites. This means that websites relying on JavaScript-rendered content may only have their unrendered HTML read by most AI crawlers, while the behavior of OpenAI's crawler has already changed on some websites.

AI overview based on source reports · Updated 9 hr ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 17
  1. Hacker News AI
    Ready or Not, Here Comes AI, by Chris Shiflett

    Chris Shiflett argues that AI crawlers like GPTBot and ClaudeBot resemble the text-only Googlebot of 2004, fetching HTML and largely not executing JavaScript, and that this makes clean, well-structured HTML newly important for being found. He describes building tools to check whether AI can find his sites and whether it actually recommends them, concluding that findability depends on basics like robots.txt, sitemaps, real HTML content and structured data, while authority comes from what others say about you.

Activity for this story

Current comparable activity
8
Peak comparable activity
10Nov 18 01:00
Change in the past 24 hours
–

Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.