mlsanityData to 7 Oct 2026

μ-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts

arXiv 2505.18451 ↗First posted 24 May 2025cs.LG cs.AI cs.CLcited by 3

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

To tackle the huge computational demand of large foundation models, activation-aware compression techniques without retraining have been introduced. However, since these rely on calibration data, domain shift may arise for unknown downstream tasks. With a computationally efficient calibration, activation-aware pruning can be executed for every prompt adaptively, yet achieving reduced complexity at inference. We formulate it as a mixture of micro-experts, called μ-MoE. Several experiments demonstrate that μ-MoE can dynamically adapt to task/prompt-dependent structured sparsity on the fly.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 46 of its 60 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

46 of 60 references.

References · 46

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
34 more
  1. 2024Mixtral of Expertscited by 1,785
  2. 2019Towards VQA Models That Can Readcited by 1,696
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  5. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  6. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  7. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  8. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  9. 2020What is the State of Neural Network Pruning?cited by 578
  10. 2023A Survey on Model Compression for Large Language Modelscited by 263
  11. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
  12. 2024Mixture of LoRA Expertscited by 179
  13. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 156
  14. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  15. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  16. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 130
  17. 2018Dynamic Channel Pruning: Feature Boosting and Suppressioncited by 129
  18. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 108
  19. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  20. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compressioncited by 71
  21. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64
  22. 2024Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Modelscited by 64
  23. 2017Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Executioncited by 59
  24. 2024Compressing Large Language Models using Low Rank and Low Precision Decompositioncited by 58
  25. 2023On the Impact of Calibration Data in Post-training Quantization and Pruningcited by 51
  26. 2024Q-VLM: Post-training Quantization for Large Vision-Language Modelscited by 47
  27. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  28. 2024LoTR: Low Tensor Rank Weight Adaptationcited by 27
  29. 2024SparseLLM: Towards Global Pruning for Pre-trained Language Modelscited by 23
  30. 2024MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuningcited by 19
  31. 2025Quantum-PEFT: Ultra parameter-efficient fine-tuningcited by 9
  32. 2024PC-LoRA: Low-Rank Adaptation for Progressive Model Compression with Knowledge Distillationcited by 8
  33. 2019Self-Adaptive Network Pruningcited by 7
  34. 2025Dynamic DropConnect: Enhancing Neural Network Robustness through Adaptive Edge Dropping Strategiescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Flycited by 1
  2. 2026POP: Online Structural Pruning Enables Efficient Inference of Large Foundation Modelscited by 0
  3. 2026EinSort: Sorting is All We Need for Tensorizing LLMcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression
  2. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compression
  3. 2025MoE-PHDS: One MoE checkpoint for flexible runtime sparsity
  4. 2026ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration
  5. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  6. 2026MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference
  7. 2026FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models
  8. 2025DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning