mlsanityData to 14 Oct 2026

Mixture of Parrots: Experts improve memorization more than reasoning

arXiv 2410.19034 ↗First posted 24 Oct 2024cs.LGcited by 21

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

The Mixture-of-Experts (MoE) architecture enables a significant increase in the total number of model parameters with minimal computational overhead. However, it is not clear what performance tradeoffs, if any, exist between MoEs and standard dense transformers. In this paper, we show that as we increase the number of experts (while fixing the number of active parameters), the memorization performance consistently increases while the reasoning capabilities saturate. We begin by analyzing the theoretical limitations of MoEs at reasoning. We prove that there exist graph problems that cannot be solved by any number of experts of a certain width; however, the same task can be easily solved by a dense model with a slightly larger width. On the other hand, we find that on memory-intensive tasks, MoEs can effectively leverage a small number of active parameters with a large number of experts to memorize the data. We empirically validate these findings on synthetic graph problems and memory-intensive closed book retrieval tasks. Lastly, we pre-train a series of MoEs and dense transformers and evaluate them on commonly used benchmarks in math and natural language. We find that increasing the number of experts helps solve knowledge-intensive tasks, but fails to yield the same benefits for reasoning tasks.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 81 of its 105 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

81 of 105 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence17 Jun
  2. 2024Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training6 May
  3. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models8 Apr
  4. 2024Scaling Laws for Fine-Grained Mixture of Experts12 Feb
  5. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  6. 2024Mixtral of Experts8 Jan
  7. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  8. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  9. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 70

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  9. 2023Mistral 7Bcited by 3,164
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
38 more
  1. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  3. 2024Qwen2 Technical Reportcited by 2,120
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  5. 2019Language Models as Knowledge Bases?cited by 1,936
  6. 2017Decoupled Weight Decay Regularizationcited by 1,831
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  9. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  10. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  11. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  12. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  13. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  14. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  15. 2022PAL: Program-aided Language Modelscited by 652
  16. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  17. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  18. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 410
  19. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  20. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
  21. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  22. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  23. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  24. 2023Can Language Models Solve Graph Problems in Natural Language?cited by 252
  25. 2021Hash Layers For Large Sparse Modelscited by 237
  26. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  27. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  28. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
  29. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
  30. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 199
  31. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 192
  32. 2021Thinking Like Transformerscited by 184
  33. 2023Talk like a Graph: Encoding Graphs for Large Language Modelscited by 176
  34. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 155
  35. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  36. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  37. 2023What Formal Languages Can Transformers Express? A Surveycited by 124
  38. 2023Teaching Arithmetic to Small Transformerscited by 123

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 10; 2026: 9.

21 papers.

Sort:

By area:

  1. 2025Distillation Scaling Lawscited by 67
  2. 2024Understanding Factual Recall in Transformers via Associative Memoriescited by 51
  3. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 50
  4. 2024Theoretical limitations of multi-layer Transformercited by 45
  5. 2026Training Language Models via Neural Cellular Automatacited by 15
  6. 2025Pretraining with hierarchical memories: separating long-tail and common knowledgecited by 14
  7. 2025The power of fine-grained experts: Granularity boosts expressivity in Mixture of Expertscited by 9
  8. 2026Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Modelscited by 6
  9. 2025Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognitioncited by 4
  10. 2025When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspectivecited by 3
  11. 2026Cram Less to Fit More: Training Data Pruning Improves Memorization of Factscited by 3
  12. 2025Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMscited by 2
9 more
  1. 2026Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimizationcited by 2
  2. 2026The Rules-and-Facts Model for Simultaneous Generalization and Memorization in Neural Networkscited by 2
  3. 2025Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Taskscited by 1
  4. 2025From Parameters to Performance: A Data-Driven Study on LLM Structure and Developmentcited by 1
  5. 2026Robustness of Mixtures of Experts to Feature Noisecited by 1
  6. 2026Optimal Splitting of Language Models from Mixtures to Specialized Domainscited by 1
  7. 2025Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspectivecited by 0
  8. 2026Selective Ensemble Based on Preference-Directed Multi-Objective Banditscited by 0
  9. 2026MoRE: Scaling mixture of experts with hardware-aware low-rank routingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  2. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
  3. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
  4. 2026You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs
  5. 2026A theoretical model for task routing in mixture-of-expert transformers
  6. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  7. 2026DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
  8. 2024DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models