mlsanityData to 7 Oct 2026

Accelerating Dense LLMs via L0-regularized Mixture-of-Experts

arXiv 2609.21672 ↗First posted 18 Sep 2026cs.AI cs.CLcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Large language models (LLMs) achieve strong performance but suffer from slow and costly inference. Existing acceleration methods often lead to noticeable performance degradation, while Mixture-of-Experts (MoE) models require extensive computational resources. In this paper, we propose L0-MoE, a lightweight MoE approach using L0-regularization to accelerate dense LLMs nearly without performance loss. Our method introduces a cluster confusion matrix for domain-aware dataset curation and applies dynamic batching for efficient training. Experiments show that L0-MoE achieves up to 2.5x speedup over dense models while maintaining competitive performance, outperforming existing LLM acceleration baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 23 of its 32 references that are in the map. First posted on arXiv 18 Sep 2026.

Builds on

23 of 32 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training24 Jun
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb

Other references · 20

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2024Qwen2 Technical Reportcited by 2,076
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
8 more
  1. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  2. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  3. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  4. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  5. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  6. 2019Structured Pruning of Large Language Modelscited by 201
  7. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  8. 2021M-FAC: Efficient Matrix-Free Approximations of Second-Order Informationcited by 54

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  2. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  3. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
  4. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  5. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains More
  6. 2025Faster MoE LLM Inference for Extremely Large Models
  7. 2026cMoLLM at Scale: Horizontal Scaling Laws for Mixture-of-LLMs
  8. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models