A GPT trained from scratch solves 99.9% of 15-puzzle boards
Understand this story
Aatish Bhatia experimented with the 15-puzzle: after training a GPT from scratch on 1 million solutions generated by a teacher algorithm, it could solve 99.9% of randomly scrambled boards. Of the 3,000 puzzles tested, 2,996 matched the teacher's moves exactly. This result is used to illustrate the "jaggedness" of AI capabilities: the same model approaches teacher-level performance on the puzzle task, but the report does not give comparative results for it on other tasks.
AI overview based on source reports · Updated 19 hr ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AISimple Puzzle Reveals AI's Jagged Intelligence
An article describes training a GPT from scratch on a million solutions to the 15 puzzle, after which it solves 99.9 percent of randomly scrambled boards but fails on the easiest puzzles, including one that is already solved, while assigning its wrong moves over 97 percent probability. The author attributes this to AI emulating its training data through shortcuts rather than understanding, producing a "jagged" boundary where performance collapses outside the training distribution. The piece draws on Ilya Sutskever and Melanie Mitchell to argue that skill is not understanding and that AI can be confidently wrong.
Activity for this story
- Current comparable activity
- 6
- Peak comparable activity
- 10Nov 14 03:00
- Change in the past 24 hours
- –
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.