Don't build your own harness: lessons from building Paywalls AI Editor

RevenueCat describes how its Paywalls AI editor initially manipulated a custom JSON paywall schema through a bespoke harness, producing results that lacked taste despite model, prompt and skill tweaks. The team switched to letting the agent edit a JSX representation of the paywall with familiar Read, Write, Edit and compiler tools, which set a record on their blank-paywalls eval two days after the prototype and let them cut over half the system prompt and almost all skills. After rollout to 100% of users on September 2nd, positive feedback across roughly 46K requests grew 83% and negative feedback fell 40%, with gains concentrated in large edits, new paywall creation and anything requiring taste.