# Yuan 2.0-M32: Mixture of Experts with Attention Router

arXiv: https://arxiv.org/abs/2405.17976 · HTML: https://mlsanity.com/paper/2405.17976/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 1 of its 2 references that are in the map.
- First posted on arXiv 28 May 2024.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 1 of 1)

- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)

## Built on by (most cited first, 18 of 18)

- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024)
- [A Survey on Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2407.06204/) (2024)
- [A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications](https://mlsanity.com/paper/2503.07137/) (2025)
- [MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router](https://mlsanity.com/paper/2410.12013/) (2024)
- [A Closer Look into Mixture-of-Experts in Large Language Models](https://mlsanity.com/paper/2406.18219/) (2024)
- [A Survey on Inference Optimization Techniques for Mixture of Experts Models](https://mlsanity.com/paper/2412.14219/) (2024)
- [SimpleLLM4AD: An End-to-End Vision-Language Model with Graph Visual Question Answering for Autonomous Driving](https://mlsanity.com/paper/2407.21293/) (2024)
- [Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models](https://mlsanity.com/paper/2505.16056/) (2025)
- [Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource](https://mlsanity.com/paper/2506.12119/) (2025)
- [GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanism](https://mlsanity.com/paper/2501.07890/) (2025)
- [LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2411.00918/) (2024)
- [Model Hemorrhage and the Robustness Limits of Large Language Models](https://mlsanity.com/paper/2503.23924/) (2025)
- [Fine-grained MoE Load Balancing with Linear Programming](https://mlsanity.com/paper/2511.16947/) (2025)
- [L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts](https://mlsanity.com/paper/2601.21349/) (2026)
- [RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism](https://mlsanity.com/paper/2605.25565/) (2026)
- [Multi-modal Collaborative Optimization and Expansion Network for Event-assisted Single-eye Expression Recognition](https://mlsanity.com/paper/2505.12007/) (2025)
- [Self-Adaptive Graph Mixture of Models](https://mlsanity.com/paper/2511.13062/) (2025)
- [Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling](https://mlsanity.com/paper/2603.27624/) (2026)

## Abstract (arXiv)

Yuan 2.0-M32, with a similar base architecture as Yuan-2.0 2B, uses a mixture-of-experts architecture with 32 experts of which 2 experts are active. A new router network, Attention Router, is proposed and adopted for a more efficient selection of experts, which improves the accuracy compared to the model with classical router network. Yuan 2.0-M32 is trained with 2000B tokens from scratch, and the training computation consumption is only 9.25% of a dense model at the same parameter scale. Yuan 2.0-M32 demonstrates competitive capability on coding, math, and various domains of expertise, with only 3.7B active parameters of 40B in total, and 7.4 GFlops forward computation per token, both of which are only 1/19 of Llama3-70B. Yuan 2.0-M32 surpass Llama3-70B on MATH and ARC-Challenge benchmark, with accuracy of 55.89 and 95.8 respectively. The models and source codes of Yuan 2.0-M32 are released at Github1.