Memdoor: A team of 5 Qwen3.8-27B models beats Fable 5 on LiveCodeBench Hard

This method of using a team of LLMs to solve complex coding problems is now available as an open-source tool.

A team of five Qwen3.8-27B language models, orchestrated by a Memdoor workflow, has achieved a near-perfect score on the LiveCodeBench Hard benchmark. This setup, comprising a planner, two ideators, a solver, and a manager, with an additional repair worker, successfully solved 49 out of 50 programming problems. The workflow operates by models communicating solely through files on disk, simulating a team-based problem-solving approach. In five passes over the first ten problems of the benchmark, the team achieved an average score of 9.8 out of 10. This performance narrowly surpassed their own specialized harness, which scored 9.6, and significantly outperformed Fable 5 alone, which scored 8.0. The entire process, including 50 runs and covering 15.3 million output tokens, cost approximately $35.79, or $38.77 with additional attempts stopped by time limits. The authors acknowledge a potential for model contamination, as the benchmark's problems date from early 2025, a period potentially covered by the model's training data.

The success of the Memdoor workflow demonstrates a powerful emergent capability in large language models when structured as a collaborative team. By assigning distinct roles and enforcing communication through disk files, the system effectively mimics human problem-solving processes, leading to performance that approaches or exceeds specialized single-model approaches. This approach also highlights the potential for LLMs to tackle complex, multi-step tasks that have previously been challenging for individual models.

The open-sourcing of this Memdoor workflow provides developers and researchers with a tangible tool to replicate and build upon these advanced AI problem-solving techniques. The low cost and high success rate suggest that such multi-agent systems could become a practical solution for automated coding and complex task execution, potentially lowering the barrier to entry for sophisticated AI applications.