An interview with Goodfire researcher Nick Cammarata, whose viral blog post claimed LLMs are easier to understand than thought, as other researchers push back

The possibility of understanding LLMs could significantly reduce the perceived risk of advanced AI systems.

Goodfire researcher Nick Cammarata has published a blog post claiming that large language models (LLMs) may be more understandable than previously believed, potentially making them easier to monitor and control. This assertion comes amidst recent incidents where AI agents have caused issues on the internet, increasing the industry's desire for better understanding and control of its technology. Cammarata's argument suggests that by applying more diligent analysis, researchers can decipher the logic within neural networks, challenging the long-held view that these systems are inherently tangled and inscrutable. He re-examined earlier research, such as studies on the Inception v1 model, and concluded that what appeared to be a neuron multitasking unrelated concepts was actually the model identifying abstract shared features, like a V-shape. While Cammarata acknowledges he does not yet fully understand LLMs, he believes complete comprehension is achievable, which would be a significant development given the scale of current AI models.

Cammarata's claim challenges the established notion of "superposition" in neural networks, where unrelated concepts are believed to be folded into single neurons, creating a "tangled knot" that is difficult to unravel. His new approach suggests that these "tangled" neurons are instead identifying abstract, common features across seemingly disparate data points, such as a V-shape present in both a car's front and a cat's face. This perspective reframes the interpretability problem from one of untangling complex structures to one of recognizing underlying patterns, potentially accelerating progress in understanding AI behavior.

If Cammarata's hypothesis holds true for current, more complex LLMs, it could lead to more robust AI safety measures and faster development of more capable AI systems. The ability to "map out whole circuitry" would allow for more precise control and a deeper understanding of emergent behaviors, reassuring both developers and the public about the safety of increasingly powerful AI. The next crucial step is the rigorous evaluation of his methods on state-of-the-art LLMs, which researchers are eagerly awaiting.