mlsanityData to 14 Oct 2026

Samoyeds: Accelerating MoE Models with Structured Sparsity Leveraging Sparse Tensor Cores

arXiv 2503.10725 ↗First posted 13 Mar 2025cs.LG cs.AI cs.DCcited by 2

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

The escalating size of Mixture-of-Experts (MoE) based Large Language Models (LLMs) presents significant computational and memory challenges, necessitating innovative solutions to enhance efficiency without compromising model accuracy. Structured sparsity emerges as a compelling strategy to address these challenges by leveraging the emerging sparse computing hardware. Prior works mainly focus on the sparsity in model parameters, neglecting the inherent sparse patterns in activations. This oversight can lead to additional computational costs associated with activations, potentially resulting in suboptimal performance. This paper presents Samoyeds, an innovative acceleration system for MoE LLMs utilizing Sparse Tensor Cores (SpTCs). Samoyeds is the first to apply sparsity simultaneously to both activations and model parameters. It introduces a bespoke sparse data format tailored for MoE computation and develops a specialized sparse-sparse matrix multiplication kernel. Furthermore, Samoyeds incorporates systematic optimizations specifically designed for the execution of dual-side structured sparse MoE LLMs on SpTCs, further enhancing system performance. Evaluations show that Samoyeds outperforms SOTA works by up to 1.99× at the kernel level and 1.58× at the model level. Moreover, it enhances memory efficiency, increasing maximum supported batch sizes by 4.41× on average. Additionally, Samoyeds surpasses existing SOTA structured sparse solutions in both model accuracy and hardware portability.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 30 of its 57 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

30 of 57 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models29 Jan
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  3. 2024Mixtral of Experts8 Jan
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  9. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  10. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  11. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  12. 2022Unified Scaling Laws for Routed Language Modelscited by 253
13 more
  1. 2021Accelerating Sparse Deep Neural Networkscited by 223
  2. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
  3. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 203
  4. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 201
  5. 2020Sparse GPU Kernels for Deep Learningcited by 141
  6. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 113
  7. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  8. 2017Exploring Sparsity in Recurrent Neural Networkscited by 75
  9. 2023PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformationcited by 25
  10. 2021NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMMcited by 19
  11. 2022Dynamic N:M Fine-grained Structured Sparse Attention Mechanismcited by 14
  12. 2023VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Corescited by 13
  13. 2022Survey: Exploiting Data Redundancy for Optimization of Deep Learningcited by 6

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
  2. 202510Cache: Heterogeneous Resource-Aware Tensor Caching and Migration for LLM Trainingcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts
  2. 2024MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems
  3. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems
  4. 2026Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity
  5. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  6. 2024Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks
  7. 2025MoE-PHDS: One MoE checkpoint for flexible runtime sparsity
  8. 2024GQSA: Group Quantization and Sparsity for Accelerating Large Language Model Inference