mlsanityData to 7 Oct 2026

ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning

arXiv 2501.15316 ↗First posted 25 Jan 2025cs.LG cs.CLcited by 10

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable abilities in tackling a wide range of complex tasks. However, their huge computational and memory costs raise significant challenges in deploying these models on resource-constrained devices or efficiently serving them. Prior approaches have attempted to alleviate these problems by permanently removing less important model structures, yet these methods often result in substantial performance degradation due to the permanent deletion of model parameters. In this work, we tried to mitigate this issue by reducing the number of active parameters without permanently removing them. Specifically, we introduce a differentiable dynamic pruning method that pushes dense models to maintain a fixed number of active parameters by converting their MLP layers into a Mixture of Experts (MoE) architecture. Our method, even without fine-tuning, consistently outperforms previous structural pruning techniques across diverse model families, including Phi-2, LLaMA-2, LLaMA-3, and Qwen-2.5.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 56 of its 70 references that are in the map. First posted on arXiv 25 Jan 2025.

Builds on

56 of 70 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Qwen2.5 Technical Report19 Dec
  2. 2024LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training24 Nov
  3. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training24 Jun
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  5. 2024Mixtral of Experts8 Jan
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2020Scaling Laws for Neural Language Modelscited by 6,411
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
37 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  11. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  13. 2016HyperNetworkscited by 1,003
  14. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  15. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  16. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  17. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  18. 2019Importance Estimation for Neural Network Pruningcited by 531
  19. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  20. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  21. 2019Dynamic Convolution: Attention over Convolution Kernelscited by 355
  22. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  23. 2023DepGraph: Towards Any Structural Pruningcited by 202
  24. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  25. 2015Structured Pruning of Deep Convolutional Neural Networkscited by 148
  26. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  27. 2018Dynamic Channel Pruning: Feature Boosting and Suppressioncited by 129
  28. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  29. 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 96
  30. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  31. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  32. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 45
  33. 2024Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Modelscited by 40
  34. 2023The LLM Surgeoncited by 36
  35. 2024DISP-LLM: Dimension-Independent Structural Pruning for Large Language Modelscited by 24
  36. 2024Learn To be Efficient: Build Structured Sparsity in Large Language Modelscited by 19
  37. 2024Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image Diffusion Modelscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generationcited by 4
  2. 2025HyperEdit: Unlocking Instruction-based Text Editing in LLMs via Hypernetworkscited by 2
  3. 2025StructPrune: Structured Global Pruning asymptotics with O(√N) GPU Memorycited by 1
  4. 2025MLPMoE: Zero-Shot Architectural Metamorphosis of Dense LLM MLPs into Static Mixture-of-Expertscited by 1
  5. 2026MoEITS: A Green AI approach for simplifying MoE-LLMscited by 1
  6. 2026Visualising Information Flow in Word Embeddings with Diffusion Tensor Imagingcited by 0
  7. 2026ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patternscited by 0
  8. 2026Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottleneckscited by 0
  9. 2026Mixture-of-Experts in Remote Sensing: A Surveycited by 0
  10. 2026Dense2MoE: Pushing the Pareto Frontier of On-Device LLMs via Unified Pruning and Upcyclingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
  2. 2025Pruning Laws for Large Language Models
  3. 2026Does a Global Perspective Help Prune Sparse MoEs Elegantly?
  4. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs
  5. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
  6. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  7. 2025MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE
  8. 2025Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Models