mlsanityData to 14 Oct 2026

Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion

arXiv 2310.04361 ↗First posted 6 Oct 2023cs.LGcited by 16

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Transformer models can face practical limitations due to their high computational requirements. At the same time, such models exhibit significant activation sparsity, which can be leveraged to reduce the inference cost by converting parts of the network into equivalent Mixture-of-Experts (MoE) layers. Despite the crucial role played by activation sparsity, its impact on this process remains unexplored. We demonstrate that the efficiency of the conversion can be significantly enhanced by a proper regularization of the activation sparsity of the base model. Moreover, motivated by the high variance of the number of activated neurons for different inputs, we introduce a more effective dynamic-k expert selection rule that adjusts the number of executed experts on a per-token basis. To achieve further savings, we extend this approach to multi-head attention projections. Finally, we develop an efficient implementation that translates these computational savings into actual wall-clock speedup. The proposed method, Dense to Dynamic-k Mixture-of-Experts (D2DMoE), outperforms existing approaches on common NLP and vision tasks, reducing inference cost by up to 60% without significantly impacting performance.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 41 of its 59 references that are in the map. First posted on arXiv 6 Oct 2023.

Builds on

41 of 59 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
29 more
  1. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  2. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  3. 2020GLU Variants Improve Transformercited by 1,804
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  5. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  6. 2021Transformers in Vision: A Surveycited by 1,013
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  8. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  9. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  10. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  11. 2021A White Paper on Neural Network Quantizationcited by 472
  12. 2022DeiT III: Revenge of the ViTcited by 419
  13. 2021Dynamic Neural Networks: A Surveycited by 408
  14. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  15. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  16. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  17. 2021Hash Layers For Large Sparse Modelscited by 237
  18. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  19. 2019Adaptively Sparse Transformerscited by 155
  20. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 152
  21. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
  22. 2021Sparse is Enough in Scaling Transformerscited by 86
  23. 2023SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformercited by 60
  24. 2022CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Predictioncited by 39
  25. 2022MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptationcited by 37
  26. 2023Emergent Modularity in Pre-trained Transformerscited by 37
  27. 2018Accelerating Convolutional Neural Networks via Activation Map Compressioncited by 34
  28. 2023AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformerscited by 24
  29. 2023Mobile V-MoEs: Scaling Down Vision Transformers via Sparse Mixture-of-Expertscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 6; 2026: 5.

16 papers.

Sort:

By area:

  1. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  2. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  3. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  4. 2025Load Balancing Mixture of Experts with Similarity Preserving Routerscited by 22
  5. 2025Hidden Dynamics of Massive Activations in Transformer Trainingcited by 10
  6. 2024HiRE: High Recall Approximate Top-k Estimation for Efficient LLM Inferencecited by 7
  7. 2025Universal Properties of Activation Sparsity in Modern Large Language Modelscited by 7
  8. 2024First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Modelscited by 6
  9. 2025Spark Transformer: Reactivating Sparsity in FFN and Attentioncited by 5
  10. 2025Sparse Brains are Also Adaptive Brains: Cognitive-Load-Aware Dynamic Activation for LLMscited by 3
  11. 2026Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs'Hallucinationscited by 2
  12. 2025Dynamic Mixture-of-Experts for Visual Autoregressive Modelcited by 1
4 more
  1. 2026Mixture-of-Experts in Remote Sensing: A Surveycited by 0
  2. 2026CLEAR-MoE: Shared-Basis Expert Extraction from Frozen Vision Transformers via Calibration-Driven Layer Selectioncited by 0
  3. 2026SparSEEty: Extracting Tokens from Sparsity-Exploiting LLM Serving Systems via Deterministic Side Channelscited by 0
  4. 2026Spectral-MoE: Heterogeneous Per-Layer Expert Utilization in Dense-to-MoE Conversioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution
  2. 2025Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis
  3. 2026Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design
  4. 2024Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsity
  5. 2025DualSparse-MoE: Coordinating Tensor/Neuron-Level Sparsity with Expert Partition and Reconstruction
  6. 2026DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks
  7. 2026Robustness of Mixtures of Experts to Feature Noise
  8. 2025Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks