Our AI Agent Executed the Same Successful SQL Query 254 Times
Funnel compared the open-weight model GLM-5.2 against Claude Opus 4.8 in a randomised A/B test on its internal data agent platform Reef, finding GLM averaged 7.8 seconds per model call versus 18.1 seconds for Opus but made 66% more tool calls (19 vs 11) and took 14% longer end-to-end. In two separate runs GLM executed the same successful SQL query 113 and 254 times, in one case repeatedly querying a tool that only exposed a 20-row preview. The author attributes much of the problem to the agent harness—nonexistent Athena tables in saved instructions, ambiguous tool contracts and missing safeguards against repetitive calls—and recommends explicit tool contracts, bounded agent behaviour and whole-system benchmarking.