mlsanityData to 14 Oct 2026

Unveiling Super Experts in Mixture-of-Experts Large Language Models

arXiv 2507.23279 ↗First posted 31 Jul 2025cs.CLcited by 17

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

In this study, we report, for the first time, the discovery and systematic investigation of a distinct subset of experts that play a pivotal role in the MoE LLMs' forward inference. These experts are prevalent in open-source MoE LLMs, and despite their extremely limited number, pruning them results in a substantial decline in model performance (e.g., prune just three out of 6,144 causes Qwen3-30B-A3B to generate repetitive and uninformative outputs).We refer to these experts as Super Experts (SEs). Our comprehensive analysis provides progressively deeper insights into SEs: (i) SEs are characterized by rare but extreme activation outliers in the output of the down_proj, which give rise to massive activations in the hidden states between decoder layers. Moreover, the distribution of SEs is model-specific, data-agnostic, and remains unaffected by post-training processes. (ii) By pruning SEs, we assess their significance across a variety of tasks, revealing their considerable impact on the model's overall performance, particularly in mathematical reasoning. (iii) We further investigate why compressing SEs exerts such a pronounced impact. We show that, in MoE LLMs, SEs serve as the primary source of the systematic outlier mechanism in Transformers, and that compressing them profoundly disrupts this process, ultimately causing the collapse of attention sinks. These findings advance the understanding of the internal dynamics of MoE LLMs, filling an important gap in the current knowledge. The code is provided in https://github.com/ZunhaiSu/Super-Experts-Profilling.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 72 of its 89 references that are in the map. First posted on arXiv 31 Jul 2025.

Builds on

72 of 89 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Qwen3 Technical Report14 May
  2. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free10 May
  3. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design9 May
  4. 2025MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance2 May
  5. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition1 Nov
  6. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router15 Oct
  7. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains More8 Oct
  8. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts12 Jul
  9. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs1 Jul
  10. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  11. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models22 Feb
  12. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
1 more
  1. 2024Mixtral of Experts8 Jan

Other references · 59

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
38 more
  1. 2023Let's Verify Step by Stepcited by 3,789
  2. 2016Pointer Sentinel Mixture Modelscited by 3,096
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2023A Survey of Large Language Modelscited by 2,576
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2024Qwen2 Technical Reportcited by 2,120
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  12. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  13. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  14. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  15. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  16. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  17. 2023Vision Transformers Need Registerscited by 871
  18. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  19. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  20. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  21. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  22. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  23. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  24. 2024Scaling and evaluating sparse autoencoderscited by 553
  25. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  26. 2019Revealing the Dark Secrets of BERTcited by 301
  27. 2024Massive Activations in Large Language Modelscited by 294
  28. 2023A Survey on Model Compression for Large Language Modelscited by 269
  29. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
  30. 2020MedDialog: Two Large-scale Medical Dialogue Datasetscited by 161
  31. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 157
  32. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  33. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141
  34. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  35. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  36. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  37. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
  38. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 15.

17 papers.

Sort:

By area:

  1. 2025LongCat-Flash Technical Reportcited by 58
  2. 2026Hyperparameter Transfer with Mixture-of-Expert Layerscited by 13
  3. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyondcited by 5
  4. 2026SnapMLA: Efficient Long-Context MLA Decoding via Hardware-Aware FP8 Quantized Pipeliningcited by 4
  5. 2026Rethinking the Role of Tensor Decompositions in Post-Training LLM Compressioncited by 4
  6. 2026Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformerscited by 3
  7. 2026SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMscited by 3
  8. 2025GPT-OSS-20B: A Comprehensive Deployment-Centric Analysis of OpenAI's Open-Weight Mixture of Experts Modelcited by 2
  9. 2026Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateauscited by 2
  10. 2026Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuningcited by 1
  11. 2026Yuan3.0 Ultra: A Trillion-Parameter Enterprise-Oriented MoE LLMcited by 0
  12. 2026CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Expertscited by 0
5 more
  1. 2026Accelerating MoE with Dynamic In-Switch Computing on Multi-GPUscited by 0
  2. 2026DBES: A Systematic Benchmark and Metric Suite for Evaluating Expert Specialization in Large-Scale MoEscited by 0
  3. 2026OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuitcited by 0
  4. 2026Frequency Is Not Sensitivity Identifying Safety-Sensitive Experts in Sparse MoE LLMcited by 0
  5. 2026Not All Drift Is a Mechanism: Tracing Prompt Adaptation in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  2. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  3. 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models
  4. 2024A Survey on Mixture of Experts in Large Language Models
  5. 2025Mixture of Experts in Large Language Models
  6. 2021Efficient Large Scale Language Modeling with Mixtures of Experts
  7. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
  8. 2025Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization