Paywalls AI editor abandons self-built harness and switches to code-based Agent
Understand this story
The RevenueCat Paywalls AI Editor team says that after abandoning its self-built harness, it switched to a code-based Agent like Claude Code to generate paywalls, with noticeably better results. The team says that with the self-built harness, the model was forced to translate design taste into JSON patch tool calls, and the generation quality was far inferior to that of a code-based Agent. Based on this, the team advises developers building their own Agents to make use of tasks, environments, and tools that models have already been extensively trained on. The new AI editor is now open to all apps.
AI overview based on source reports · Updated 2 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIDon't build your own harness: lessons from building Paywalls AI Editor
RevenueCat describes how its Paywalls AI editor initially manipulated a custom JSON paywall schema through a bespoke harness, producing results that lacked taste despite model, prompt and skill tweaks. The team switched to letting the agent edit a JSX representation of the paywall with familiar Read, Write, Edit and compiler tools, which set a record on their blank-paywalls eval two days after the prototype and let them cut over half the system prompt and almost all skills. After rollout to 100% of users on September 2nd, positive feedback across roughly 46K requests grew 83% and negative feedback fell 40%, with gains concentrated in large edits, new paywall creation and anything requiring taste.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 10Nov 12 23:00
- Change in the past 24 hours
- -50%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.