Moving our warehouse off DuckDB

The cost of running your data warehouse operations will now be more predictable and potentially lower.

PostHog is moving its data warehouse infrastructure away from DuckDB, just five months after migrating to it from ClickHouse. The company initially adopted DuckDB to provide each customer with their own instance and improve compatibility with tools like dbt via Postgres connections, addressing issues with a shared ClickHouse cluster's performance and lack of a cost-based query optimizer. However, PostHog found that DuckDB, designed for single-node performance, struggled with the growing complexity and scale of customer workloads in a cloud environment. Challenges included managing resources for unpredictable queries, performance bottlenecks with the DuckLake extension for handling large datasets (up to 2 PB or 1 trillion rows), and difficulties scaling Postgres connections. PostHog is now rebuilding its warehouse on Trino, using a new open table format called HogLake, to better handle petabyte-scale data and provide a superior customer experience.

The migration away from DuckDB signals a broader market trend where single-node, though powerful, data processing engines face inherent scalability limitations when deployed at cloud-scale for multi-tenant environments. Companies like PostHog are discovering that while such engines excel in local analysis or smaller deployments, the complexities of distributed systems, resource management, and optimization at petabyte scale necessitate architectures designed for distribution from the ground up.

This move impacts companies relying on managed warehouse solutions that may have adopted similar single-node architectures. It suggests a need for users to scrutinize the underlying technology powering their data warehouses, especially if their data volumes or query complexity are growing rapidly. PostHog's adoption of Trino with their custom HogLake format indicates a preference for open, distributed query engines and extensible table formats as a path forward for high-scale data operations.