mlsanityData to 7 Oct 2026

Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity

arXiv 2305.02176 ↗First posted 3 May 2023cs.CLcited by 3

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-experts (MoE) models that employ sparse activation have demonstrated effectiveness in significantly increasing the number of parameters while maintaining low computational requirements per token. However, recent studies have established that MoE models are inherently parameter-inefficient as the improvement in performance diminishes with an increasing number of experts. We hypothesize this parameter inefficiency is a result of all experts having equal capacity, which may not adequately meet the varying complexity requirements of different tokens or tasks. In light of this, we propose Stratified Mixture of Experts (SMoE) models, which feature a stratified structure and can assign dynamic capacity to different tokens. We demonstrate the effectiveness of SMoE on three multilingual machine translation benchmarks, containing 4, 15, and 94 language pairs, respectively. We show that SMoE outperforms multiple state-of-the-art MoE models with the same or fewer parameters.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 31 of its 36 references that are in the map. First posted on arXiv 3 May 2023.

Builds on

31 of 36 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022No Language Left Behind: Scaling Human-Centered Machine Translation11 Jul
  2. 2022Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers28 May
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  4. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference24 Sep
  5. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models22 Sep
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2016Layer Normalizationcited by 5,579
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  12. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
12 more
  1. 2020On Layer Normalization in the Transformer Architecturecited by 906
  2. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  3. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  4. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  6. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  7. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  8. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  9. 2019Massively Multilingual Neural Machine Translationcited by 246
  10. 2021Hash Layers For Large Sparse Modelscited by 236
  11. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  12. 2022Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Modelscited by 31

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2.

3 papers.

  1. 2024Progress and Opportunities of Foundation Models in Bioinformaticscited by 13
  2. 2023Condensing Multilingual Knowledge with Lightweight Language-Specific Modulescited by 3
  3. 2024MultiMUC: Multilingual Template Filling on MUC-4cited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2024Investigating the potential of Sparse Mixtures-of-Experts for multi-domain neural machine translation
  2. 2022AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models
  4. 2025MLPMoE: Zero-Shot Architectural Metamorphosis of Dense LLM MLPs into Static Mixture-of-Experts
  5. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
  6. 2024Retraining-Free Merging of Sparse MoE via Hierarchical Clustering
  7. 2024HMoE: Heterogeneous Mixture of Experts for Language Modeling
  8. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs