Microsoft unveils MAI-Base-2, a MoE model with 2.8T total and 80B active parameters, claiming 4x-12x more pretraining FLOPs-efficiency than DeepSeek-V4-Pro-Base

Microsoft's MAI-Base-2 model is now 4x-12x more efficient in pre-training FLOPs, lowering the cost of training large language models.

Microsoft has unveiled MAI-Base-2, a new Mixture-of-Experts (MoE) foundation model. This model boasts 2.8 trillion total parameters and 80 billion active parameters. Microsoft AI claims that MAI-Base-2 is significantly more efficient in pre-training compared to existing open-weight models, specifically citing a 4x to 12x improvement in FLOPs-efficiency over DeepSeek-V4-Pro-Base. This enhanced efficiency allowed for the training of a larger model by reinvesting the saved computational resources. The company states that MAI-Base-2 outperforms all tested open-weight models on their internal evaluations across various pre-training tasks, indicating a leap in performance for their proprietary AI development.

The development of MAI-Base-2 highlights a significant advancement in optimizing large language model pre-training. By achieving substantial FLOPs-efficiency gains, Microsoft is effectively reducing the computational resources and, by extension, the cost required to train state-of-the-art AI models. This breakthrough could enable more rapid iteration and development cycles for proprietary AI systems within Microsoft, and potentially influence how other organizations approach their own model training strategies.

This increased efficiency directly translates to a competitive advantage, allowing Microsoft to develop more powerful models without a proportional increase in training expenditure. The success of MAI-Base-2's pre-training efficiency suggests a promising direction for future AI hardware and software co-design, where architectural innovations are geared towards maximizing computational throughput and minimizing energy consumption for training massive neural networks.