I'm sorry, but you still have to think

AI-generated code may not adhere to implicit requirements, requiring human oversight for critical decisions and trade-offs.

DHH, creator of Ruby on Rails, has prompted AI to rewrite his application, Campfire Once, in Rust, Elixir, and Go. The AI-generated rewrites have exposed significant differences in implementation and performance, highlighting challenges in relying solely on AI for complex coding tasks. The Rust version, for instance, dropped full backwards compatibility to improve caching and replaced Redis with in-process queues. The Elixir rewrite was closer to the original but exhibited performance issues, such as a single process handling all SQL queries sequentially. Further analysis revealed non-async database operations and suboptimal lock usage in the Rust version, impacting its ability to yield to the runtime and leading to performance degradation. Benchmarking attempts also proved problematic, with one test showing a 1% success rate for new post notifications in the Rust version under heavy load, compared to 100% in Elixir. Subsequent re-tests with adjusted parameters and different testing methodologies revealed further trade-offs in error handling, latency, and memory usage across the different language implementations.

The AI's handling of implicit requirements, such as backwards compatibility, latency versus throughput, and acceptable error rates, demonstrates a critical gap in current AI coding capabilities. Without explicit prompting on these nuanced trade-offs, AI defaults can lead to implementations that, while functional, may not align with desired system behavior or performance profiles. This underscores the necessity for human programmers to meticulously define these constraints, as the AI lacks the contextual understanding to make these critical decisions independently.

The analysis of the Rust and Elixir rewrites reveals that benchmark results, especially those focusing on throughput, can be misleading without considering other system properties like reliability and user experience. The varying success rates and latencies in notification delivery, coupled with different error handling strategies (e.g., client disconnection versus slow delivery), highlight that optimal solutions involve careful selection and tuning of components like broadcast channels and their capacities. This indicates that developers must still possess deep domain knowledge to interpret AI outputs and configure systems effectively for their specific needs.