Fixing Tool-Call Accuracy in vLLM

Agentic AI applications now execute tool calls with 100% schema accuracy.

Developers Jesse Donkervliet and Junhao Li have addressed tool-call accuracy issues in vLLM, an inference engine for large language models. They observed that open-weight models, when served using vLLM version 0.28, failed to adhere to tool-call schemas up to 8% of the time. This inaccuracy leads to poor quality results, particularly in agentic applications requiring multi-turn conversations. The problem arises when LLMs are instructed to use external tools and must output structured data according to predefined schemas. vLLM version 0.31 now includes a server-side flag to enable constrained decoding, a technique that guarantees adherence to these schemas. This feature, when enabled, has been shown to achieve 100% schema accuracy with newer models like Kimi and DeepSeek.

The introduction of constrained decoding in vLLM 0.31 is a significant advancement for LLM-powered agentic systems. Previously, inaccuracies in tool-call formatting led to contextual errors and degraded performance, especially in conversational AI. By ensuring 100% schema adherence, this update directly enhances the reliability and output quality of models engaging in complex, multi-step tasks that rely on external tool integration.

This development impacts developers building AI agents and applications that require interaction with external tools or APIs. The previous inconsistency meant that developers had to build in significant error handling and retry mechanisms, or accept a lower baseline of performance. With guaranteed schema accuracy, the development of more robust and sophisticated AI agents becomes more feasible, reducing the burden on developers and improving the user experience.