Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
AI Signal Decode
The core innovation of Deltafin lies in its ability to stream the massive 2.8T Kimi K3 model from SSDs to a MacBook Pro. This bypasses the need for immense, dedicated VRAM by leveraging high-speed local storage. Achieving around 1 token per second on consumer hardware for a model of this scale is a notable technical feat, especially when compared to the much slower speeds reported previously for the same model on similar setups without this streaming optimization. The use of four SSDs is critical, suggesting a sophisticated I/O management strategy to feed the model data fast enough.
Market implications are substantial for democratizing access to frontier AI models. By enabling powerful models like Kimi K3 to run on setups costing thousands of dollars rather than millions, Deltafin could foster a new wave of self-hosted AI applications and research. This could empower individual developers, smaller research teams, and even hobbyists to experiment with and deploy state-of-the-art LLMs without relying on cloud providers or massive data centers, potentially shifting the landscape of AI development and accessibility.
Technically, Deltafin's approach highlights advancements in efficient model loading and inference for extremely large, Mixture-of-Experts (MoE) models. The project explicitly states it avoids pruning or skipping any of Kimi K3's 16 experts, ensuring full model fidelity. This contrasts with methods that reduce model size through quantization or expert selection, which can impact output quality. The success of streaming from SSDs suggests that I/O throughput and latency are becoming increasingly manageable bottlenecks for LLM inference, especially when combined with optimized software like Deltafin.
Future developments to watch include further optimizations for speed and latency, particularly in the 'time to first token' metric. The project's scaling tests with varying numbers of SSDs (one, two, and three drives yielding progressively better results) indicate potential for even greater performance gains with hardware improvements or more advanced storage configurations. It will also be important to see if this approach can be extended to other massive MoE models and how the performance scales as models continue to grow in parameter count and complexity.