Retrospectively Reverse-Engineering Apple's Neural Engine

The architecture of Apple's standalone Neural Engine is becoming obsolete and its functionality is being integrated into GPUs.

Eileen Yoon has retrospectively reverse-engineered Apple's Neural Engine (ANE) on the M1 chip, detailing its internal architecture three years after initially working on the ANE driver. Yoon's analysis reveals the ANE's compute cores contain 128 FP16 or 256 INT8 parallel multiply-accumulate (MAC) lanes, totaling 2048 lanes across the 16 cores. The architecture was designed for efficient dataflow, particularly for CNN workloads prevalent in 2017, with MAC operations accumulating sums locally to avoid external memory access. The ANE supports fused layers where MAC sums feed directly into pointwise activation functions like TANH, implemented via a 33-entry piecewise-linear lookup table with interpolation. Yoon's work also highlights how scaling and bias operations can be folded into the convolution and activation stages at compile time. This analysis comes as Apple integrates ANE cores into GPU cores in its M5 chip, signaling a potential decline for standalone NPUs.

The reverse-engineering effort provides a detailed look into the ANE's hardware design, revealing how it specialized in dense CNN workloads through its MAC units and dataflow. The local accumulation within MAC lanes and fused activation functions were key to its efficiency on mobile devices for tasks like image processing. This deep dive into the silicon's design choices shows Apple's early assumptions about machine learning workloads, which heavily favored predictable reuse patterns. The integration of ANE cores into the M5 chip's GPU signifies a strategic shift by Apple, acknowledging that the specialized NPU architecture is less relevant for current transformer-based workloads. While the core compute capabilities might still be useful, their enclosure within broader GPU architectures suggests a move towards more general-purpose acceleration. This transition implies that future ML acceleration on Apple Silicon will likely leverage the flexibility of GPUs over dedicated NPUs.