mlsanityData to 14 Oct 2026

Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts

arXiv 2508.07785 ↗First posted 11 Aug 2025cs.CLcited by 9

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

The Mixture of Experts (MoE) architecture is a cornerstone of modern state-of-the-art (SOTA) large language models (LLMs). MoE models facilitate scalability by enabling sparse parameter activation. However, traditional MoE architecture uses homogeneous experts of a uniform size, activating a fixed number of parameters irrespective of input complexity and thus limiting computational efficiency. To overcome this limitation, we introduce Grove MoE, a novel architecture incorporating experts of varying sizes, inspired by the heterogeneous big.LITTLE CPU architecture. This architecture features novel adjugate experts with a dynamic activation mechanism, enabling model capacity expansion while maintaining manageable computational overhead. Building on this architecture, we present GroveMoE-Base and GroveMoE-Inst, 33B-parameter LLMs developed by applying an upcycling strategy to the Qwen3-30B-A3B-Base model during mid-training and post-training. GroveMoE models dynamically activate 3.14-3.28B parameters based on token complexity and achieve performance comparable to SOTA open-source models of similar or even larger size.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 37 of its 44 references that are in the map. First posted on arXiv 11 Aug 2025.

Builds on

37 of 44 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Qwen3 Technical Report14 May
  2. 2025Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization26 Feb
  3. 2024Qwen2.5 Technical Report19 Dec
  4. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent4 Nov
  5. 2024Mixtral of Experts8 Jan
  6. 2024Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks5 Jan
  7. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec

Other references · 30

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  10. 2025Gemma 3 Technical Reportcited by 1,824
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
18 more
  1. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  2. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  3. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  4. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  5. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  6. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  7. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  8. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  9. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  10. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 284
  11. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  12. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Modelscited by 72
  13. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 56
  14. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 56
  15. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 54
  16. 2024AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Modelscited by 44
  17. 2025Parallel Scaling Law for Language Modelscited by 30
  18. 2023Alternating Updates for Efficient Transformerscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2025Mixture of Neuron Expertscited by 7
  2. 2025Route Experts by Sequence, not by Tokencited by 6
  3. 2026Post-Trained MoE Can Skip Half Experts via Self-Distillationcited by 4
  4. 2025Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Modelscited by 3
  5. 2026SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMscited by 2
  6. 2025Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Expertscited by 1
  7. 2026Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Expertscited by 1
  8. 2026Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scalingcited by 0
  9. 2026TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
  2. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  3. 2025Faster MoE LLM Inference for Extremely Large Models
  4. 2025Mixture of Experts in Large Language Models
  5. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
  6. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading
  7. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
  8. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models