mlsanityData to 14 Oct 2026

Yuan 2.0-M32: Mixture of Experts with Attention Router

arXiv 2405.17976 ↗First posted 28 May 2024cs.AI cs.CLcited by 18

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Yuan 2.0-M32, with a similar base architecture as Yuan-2.0 2B, uses a mixture-of-experts architecture with 32 experts of which 2 experts are active. A new router network, Attention Router, is proposed and adopted for a more efficient selection of experts, which improves the accuracy compared to the model with classical router network. Yuan 2.0-M32 is trained with 2000B tokens from scratch, and the training computation consumption is only 9.25% of a dense model at the same parameter scale. Yuan 2.0-M32 demonstrates competitive capability on coding, math, and various domains of expertise, with only 3.7B active parameters of 40B in total, and 7.4 GFlops forward computation per token, both of which are only 1/19 of Llama3-70B. Yuan 2.0-M32 surpass Llama3-70B on MATH and ARC-Challenge benchmark, with accuracy of 55.89 and 95.8 respectively. The models and source codes of Yuan 2.0-M32 are released at Github1.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 1 of its 2 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

1 of 2 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Mixture-of-Experts with Expert Choice Routing18 Feb

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 8; 2026: 3.

18 papers.

Sort:

By area:

  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  2. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  3. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  4. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 62
  5. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 48
  6. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  7. 2024SimpleLLM4AD: An End-to-End Vision-Language Model with Graph Visual Question Answering for Autonomous Drivingcited by 16
  8. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  9. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resourcecited by 8
  10. 2025GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanismcited by 7
  11. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  12. 2025Model Hemorrhage and the Robustness Limits of Large Language Modelscited by 1
6 more
  1. 2025Fine-grained MoE Load Balancing with Linear Programmingcited by 1
  2. 2026L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Expertscited by 1
  3. 2026RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanismcited by 1
  4. 2025Multi-modal Collaborative Optimization and Expansion Network for Event-assisted Single-eye Expression Recognitioncited by 0
  5. 2025Self-Adaptive Graph Mixture of Modelscited by 0
  6. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Schedulingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
  2. 2024Layerwise Recurrent Router for Mixture-of-Experts
  3. 2026MoRE: Scaling mixture of experts with hardware-aware low-rank routing
  4. 2025Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcycling
  5. 2026Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
  6. 2026Redesign Mixture-of-Experts Routers with Manifold Power Iteration
  7. 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
  8. 2024GW-MoE: Resolving Uncertainty in MoE Router with Global Workspace Theory