AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test

If you use AI to help write code, you will still need to manually review and fix its output for most tasks.

A new benchmark called Co-Design Bench, developed by researchers from Alibaba and several Asian universities, reveals that even the best AI coding agents only fully complete about 26.4% of real-world tasks. The benchmark tests agents on rework tasks for existing web applications based on user requests, often involving multimodal inputs like images and audio. GPT-6 Astra, the top performer, achieves about 70% of task requirements on average but falls short of full completion. Older benchmarks are criticized for being too easy, with models achieving near-perfect scores, whereas Co-Design Bench uses more complex, real-world user change requests that are more difficult for agents. Performance degrades significantly when tasks involve multiple types of changes, such as layout, design, and content, which are common in practical projects. The study also highlights the crucial role of the testing environment, showing that the same model can perform vastly differently depending on the harness used.

The development of Co-Design Bench signifies a critical shift towards evaluating AI coding agents on more realistic, complex tasks, moving beyond the saturation of existing, less challenging benchmarks. This new standard highlights the gap between theoretical performance and practical utility, as current agents struggle with the multifaceted nature of real-world coding requests. Consequently, the industry must now focus on developing agents capable of handling intricate, multimodal inputs and a combination of change types, rather than solely optimizing for simplified problem sets.

The benchmark's findings underscore that the effectiveness of AI coding agents is highly dependent on their execution environment, suggesting that reported performance metrics should always include details about the harness used. This has direct implications for developers and businesses selecting AI tools, as they must now consider the interaction between the model and its surrounding software infrastructure. Future work will likely involve improving agent robustness across diverse environments and addressing the remaining deficiencies in complex, multi-stage task completion.