mlsanityData to 14 Oct 2026

CoSMoEs: Compact Sparse Mixture of Experts

arXiv 2503.00245 ↗First posted 28 Feb 2025cs.LG cs.CLcited by 3

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Sparse Mixture of Expert (MoE) models are popular foundational architectures at large scale, however, under-explored at smaller sizes. Here, we show how to enable Compact Sparse Mixture of Experts (CoSMoEs) for on-device inference. Specifically, we tackle the three main on-device dimensions: Quality, Memory and Latency. Along the quality axis, we show that in a fair evaluation (removing confounding factors) MoE architectures outperform FLOP-aligned dense models at on-device scale. We introduce weight-decomposed experts, further improving the MoE model performance. Regarding model memory and latency, we significantly improve model offloading efficiency and, in turn, reduce model inference latency.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 21 of its 26 references that are in the map. First posted on arXiv 28 Feb 2025.

Builds on

21 of 26 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  2. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices28 Aug

Other references · 19

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  9. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  10. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  11. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  12. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
7 more
  1. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  2. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82
  3. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Expertscited by 62
  4. 2023Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penaltycited by 57
  5. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 48
  6. 2024MobiLlama: Towards Accurate and Lightweight Fully Transparent GPTcited by 34
  7. 2024BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Datacited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Short Data, Long Context: Distilling Positional Knowledge in Transformerscited by 0
  2. 2026Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligencecited by 0
  3. 2026An Exploratory Ablation of a Small MLA--SSM Hybrid Language Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models
  2. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  3. 2024MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems
  4. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems
  5. 2025Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts
  6. 2026MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference
  7. 2025MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts
  8. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models