Static

Distributed, high-performance blob storage on SQLite

First reported by Zerodha.tech ·

The signal ●○○○ Compiled by AI from Zerodha.tech and Hacker News
Why you might care

A new open-source distributed blob store built on SQLite now offers Redis protocol compatibility and high performance for multi-megabyte data.

What happened

Zerodha's engineering team developed Blobasaur, an open-source distributed blob storage server that utilizes SQLite as its underlying storage engine. This system was created to address high-throughput read/write demands for user preferences and metadata, primarily JSON blobs ranging from kilobytes to megabytes, on their trading platform, Kite. Previously, Zerodha used CouchDB and later SSDB, but these solutions presented scaling limitations and maintenance risks. Blobasaur is built in Rust, speaks the Redis protocol for easy integration with existing services, and is designed for horizontal scalability. It employs sharding across multiple SQLite databases on a single machine to achieve write concurrency, leveraging group commits for efficiency. In production, Blobasaur handles thousands of requests per second with single-digit millisecond latency on commodity hardware, effectively replacing the unmaintained SSDB.

What it means

The development of Blobasaur signals a growing trend of leveraging specialized, often open-source, solutions for specific high-performance data storage needs, moving beyond monolithic or general-purpose databases. SQLite, traditionally known for embedded applications, is being repurposed here for distributed, high-throughput scenarios, demonstrating its versatility and the potential for optimizing widely-used technologies for novel use cases. This approach challenges the notion that high-performance distributed storage requires complex, large-scale systems, offering a simpler, more economical alternative for certain workloads.

Companies facing similar challenges with user-generated metadata, UI configurations, or other non-critical, high-volume blob storage can now consider Blobasaur as a viable, cost-effective replacement for more complex solutions. The Redis protocol compatibility is a significant advantage for existing infrastructure, reducing the barrier to adoption. The project's success highlights the potential for innovation by combining mature, reliable technologies like SQLite with modern development practices and distributed system architecture principles to achieve demanding performance targets.

AI-written summary. May contain errors.

Distributed