Fancy harnesses make coding agents more expensive but not better, study finds

The cost to run coding agents can triple without improving their performance.

A new study from researchers at Shandong University has found that fancy coding agent harnesses, designed to improve performance, often only increase costs without a significant boost in results. The researchers tested three harnesses – Claude Code, OpenCode, and mini-SWE-agent – across five different language models on tasks from the SWE-bench Verified benchmark. They discovered that swapping harnesses resulted in performance changes comparable to simply rerunning the same setup, with differences often falling within the margin of measurement noise. While some harnesses, like OpenCode, showed a slight deficit, the primary impact of more elaborate harnesses was a substantial increase in token usage for system prompts and tool descriptions, tripling costs for some tasks. The study suggests that the perceived gains from advanced harnesses may be overstated, and the choice of harness has a minimal impact on success rates compared to the underlying language model itself.

The study challenges the value proposition of feature-rich coding agent harnesses by demonstrating that their purported performance gains are often indistinguishable from random noise or simple reruns. This suggests that vendors may be overemphasizing harness complexity, leading users to incur significantly higher costs for minimal or no functional improvement. The researchers advocate for standardized comparison methods, like requiring reruns alongside new harness evaluations, to bring transparency to performance metrics and prevent misleading claims in the rapidly evolving AI agent market.

This finding implies a potential shift in how AI coding tools are benchmarked and commercialized, with a greater emphasis likely to be placed on the base language model's capabilities rather than the intricate frameworks surrounding it. Users should be cautious of advertised score increases, as the study indicates these may not translate to tangible benefits and could instead represent inflated operational expenses. Future development may focus on cost-effective solutions that leverage existing models rather than solely on complex harness engineering.