Martin Fowler website proposes "tests as data" to constrain AI coding
Understand this story
An article on the Martin Fowler website proposes using "tests as data" to address the problem that AI-generated tests are untrustworthy: write inputs, expected outputs, and test states into YAML, CSV, or text scenario files, and have an AI-generated test runner execute them. Based on interviews with 26 developers, the author says AI-generated tests are numerous, hard to read, coupled to the implementation, and get changed during refactoring, so the article recommends prioritizing customer tests, black-box tests, and external DSLs, together with mutation testing, property-based testing, and manual exploratory testing. The article's stated goal is to make tests able to predict success in production, resist refactoring, and be as readable as possible.
AI overview based on source reports · Updated 1 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AITests as data: Ensuring AI-generated code does what we want
The author reports that developers at Thoughtworks distrust their tests because AI coding agents refactor both code and tests as features are added, and argues AI-generated tests are often voluminous, hard to read, coupled to implementation via mocks, and may confirm current behavior including bugs. He proposes customer tests written as data in a DSL (tables, structured scenarios, textual projections of UI and state) interpreted by a small test runner that AI can generate, drawing on Fit, Specification by Example, Go's tabular tests, and Approval Testing.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 10Nov 13 00:00
- Change in the past 24 hours
- -49%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.