mlsanityData to 14 Oct 2026

Advancing Expert Specialization for Better MoE

arXiv 2505.22323 ↗First posted 28 May 2025cs.CLcited by 39

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-Experts (MoE) models enable efficient scaling of large language models (LLMs) by activating only a subset of experts per input. However, we observe that the commonly used auxiliary load balancing loss often leads to expert overlap and overly uniform routing, which hinders expert specialization and degrades overall performance during post-training. To address this, we propose a simple yet effective solution that introduces two complementary objectives: (1) an orthogonality loss to encourage experts to process distinct types of tokens, and (2) a variance loss to encourage more discriminative routing decisions. Gradient-level analysis demonstrates that these objectives are compatible with the existing auxiliary loss and contribute to optimizing the training process. Experimental results over various model architectures and across multiple benchmarks show that our method significantly enhances expert specialization. Notably, our method improves classic MoE baselines with auxiliary loss by up to 23.79%, while also maintaining load balancing in downstream tasks, without any architectural modifications or additional components. We will release our code to contribute to the community.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 62 of its 83 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

62 of 83 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024A Survey on Mixture of Experts in Large Language Models26 Jun
  2. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training24 Jun
  3. 2024Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts17 Jun
  4. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models3 Jun
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  6. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models8 Apr
  7. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models22 Feb
  8. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning17 Feb
  9. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models29 Jan
  10. 2024LocMoE: A Low-Overhead MoE for Large Language Model Training25 Jan
  11. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  12. 2024Mixtral of Experts8 Jan
6 more
  1. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer15 Oct
  2. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models24 May
  3. 2022AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation14 Oct
  4. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  5. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun

Other references · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2021Program Synthesis with Large Language Modelscited by 3,637
  12. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
32 more
  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  9. 2018Neural Network Acceptability Judgmentscited by 958
  10. 2020Aligning AI With Shared Human Valuescited by 723
  11. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  12. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  13. 2022Efficiently Scaling Transformer Inferencecited by 539
  14. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  15. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
  16. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 203
  17. 2022A Review of Sparse Expert Models in Deep Learningcited by 148
  18. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  19. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 104
  20. 2023Symbol tuning improves in-context learning in language modelscited by 75
  21. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  22. 2024MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Modelscited by 56
  23. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 56
  24. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 55
  25. 2023Multi-Task Reinforcement Learning with Mixture of Orthogonal Expertscited by 49
  26. 2024AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Modelscited by 44
  27. 2023SiRA: Sparse Mixture of Low Rank Adaptationcited by 32
  28. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 32
  29. 2024Mixture of Nested Experts: Adaptive Processing of Visual Tokenscited by 25
  30. 2024AlphaLoRA: Assigning LoRA Experts Based on Layer Training Qualitycited by 20
  31. 2024Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Modelscited by 14
  32. 2025DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Modelscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 32.

39 papers.

Sort:

By area:

  1. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 18
  2. 2025Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Losscited by 11
  3. 2025Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectivescited by 10
  4. 2025ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specializationcited by 7
  5. 2026Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specializationcited by 5
  6. 2026φ-Balancing for Mixture-of-Experts Trainingcited by 5
  7. 2026PROBE: Co-Balancing Computation and Communication in MoE Inference via Real-Time Predictive Prefetchingcited by 4
  8. 2026Towards Intrinsic Interpretability of Large Language Models:A Survey of Design Principles and Architecturescited by 4
  9. 2026EMO: Pretraining Mixture of Experts for Emergent Modularitycited by 4
  10. 2026Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Expertscited by 4
  11. 2026Variational Inference, Entropy, and Orthogonality: A Unified Theory of Mixture-of-Expertscited by 3
  12. 2026SD-MoE: Spectral Decomposition for Effective Expert Specializationcited by 3
27 more
  1. 2025Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routingcited by 2
  2. 2026EMoE: Eigenbasis-Guided Routing for Mixture-of-Expertscited by 2
  3. 2026MagicAgent: Towards Generalized Agent Planningcited by 2
  4. 2026Compressed Sensing for Capability Localization in Large Language Modelscited by 2
  5. 2025Fine-grained MoE Load Balancing with Linear Programmingcited by 1
  6. 2026Multi-Modal Time Series Prediction via Mixture of Modulated Expertscited by 1
  7. 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scalecited by 1
  8. 2026AME-TS: Anchored Mixture-of-Experts for Time Series Forecastingcited by 1
  9. 2026Toward Calibrated Mixture-of-Experts Under Distribution Shiftcited by 1
  10. 2026LiMoDE: Rethinking Lifelong Robot Manipulation from a Mixture-of-Dynamic-Experts Perspectivecited by 1
  11. 2025Hybrid Quantum-Classical Mixture of Experts: Unlocking Topological Advantage via Interference-Based Routingcited by 0
  12. 2026Geometry-Preserving Aggregation for Mixture-of-Experts Embedding Modelscited by 0
  13. 2026Do Domain-specific Experts exist in MoE-based LLMs?cited by 0
  14. 2026Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcyclingcited by 0
  15. 2026Geometric Metrics for MoE Specialization: From Fisher Information to Early Failure Detectioncited by 0
  16. 2026Transformer as an Euler Discretization of Score-based Variational Flowcited by 0
  17. 2026Ortho-Hydra: Orthogonalized Experts for DiT LoRAcited by 0
  18. 2026E = T*H/(O+B): A Dimensionless Control Parameter for Mixture-of-Experts Ecologycited by 0
  19. 2026Variational Proximal Policy Optimizationcited by 0
  20. 2026Learning Subset-Shared Invariances for Domain Generalization with Mixture-of-Expertscited by 0
  21. 2026TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptationcited by 0
  22. 2026Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computationcited by 0
  23. 2026Router Prior Bias: Preserving Base Routing Structure in MoE Post-Trainingcited by 0
  24. 2026Mo' Models, Mo' Problems: How to best select model pools when designing Multi-Agent Systemscited by 0
  25. 2026Function Over Form: Distributional Orthogonalization in Mixture-of-Experts with Replica Expert Mechanismcited by 0
  26. 2026CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Trainingcited by 0
  27. 2026Sparse Transformer: a Controlled Testbed for Mechanistic Interpretabilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixture of Experts in Large Language Models
  2. 2026Expert Divergence Learning for MoE-based Language Models
  3. 2026Slicing and Dicing: Configuring Optimal Mixtures of Experts
  4. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  5. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  6. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  7. 2024Scaling Laws for Fine-Grained Mixture of Experts
  8. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models