What is the best harness for each model?
First reported by Runta ·
The best-value harness for your AI agent is now model-dependent, meaning you may need to switch harnesses to optimize cost and performance.
RUNTA released FrontierHarness Eval v1.1, a benchmark comparing 22 agent harnesses across five large language models. The evaluation, conducted over 27,280 agent runs using 83 billion tokens, aimed to determine if switching agent harnesses is necessary when changing models and to identify the best-value harness for each model. The study found that the optimal harness often changes with the model. For instance, Codex CLI was best with GLM-5.3, but ranked much lower with Kimi-K3. The results indicate significant cost variations, with the median harness shifting 11 places in cost rankings when models were swapped. The best-value harness was defined as one with the lowest cost per pass, or one within 30% higher cost that achieved at least 5% more points.
The research debunks the assumption that a harness's performance or cost-effectiveness remains constant across different AI models. Significant cost rank movements, with the median harness changing 11 places between model evaluations, highlight the intricate interplay between specific models and their associated harness configurations. This suggests that developers must actively re-evaluate their harness choices to achieve optimal performance and economic efficiency as they iterate on or switch between foundation models.
The study provides a concrete framework for selecting harnesses based on a best-value metric, which balances cost per pass against pass rate. This nuanced approach moves beyond simple cost-per-task metrics to identify solutions that are not only affordable but also highly effective. The varied outcomes across models underscore the evolving nature of the AI agent ecosystem, emphasizing the need for continuous benchmarking and adaptation by developers and researchers alike.
AI-written summary. May contain errors.