RTK reports token savings, but our cost benchmarks disagree

AI coding tools may not reduce your token costs as much as advertised.

A recent analysis by JetBrains has challenged claims that RTK (Rust Token Killer), a tool designed to filter and compress terminal output for AI agents, significantly reduces AI coding costs. While RTK's developers report substantial token savings, claiming up to a 60% reduction in tokens for some AI models, JetBrains's own benchmarks using the Terminal-Bench 2.1 dataset and spending over $1,500 on tokens, showed minimal or even increased costs. Specifically, the tests with Claude Code (Fable 5.0) showed a 5% cost reduction, while costs rose by 5% for OpenCode (DeepSeek V4 Pro 0813). Furthermore, RTK usage led to a slight decrease in task pass rates. The analysis suggests that RTK's reported savings, based on raw output reduction (rtk gain), do not directly translate to lower AI bills because they do not account for increased turns or potential quality degradation. One specific task even showed RTK increasing costs significantly for DeepSeek.

The core issue appears to be how RTK measures savings versus how AI models actually incur costs. RTK's 'rtk gain' metric quantifies reduced terminal output size, but this does not correlate directly with token costs, as AI agents can take more turns to achieve the same result, negating any compression benefits. This disconnect means that tools optimizing for raw output size might inadvertently lead to higher overall AI expenditure if they increase the number of interactions required.

This analysis suggests that the efficiency gains from tools like RTK are diminishing with more advanced AI models, which already incorporate methods to manage terminal output effectively. Furthermore, the specific architecture of AI platforms, where certain terminal operations bypass compression tools while cached reads are cheaper, means that direct cost savings from terminal output compression might be a niche optimization rather than a general solution for reducing AI coding expenses.