mlsanityData to 7 Oct 2026

AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation

arXiv 2210.07535 ↗First posted 14 Oct 2022cs.CL cs.LGcited by 9

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-Expert (MoE) models have obtained state-of-the-art performance in Neural Machine Translation (NMT) tasks. Existing works in MoE mostly consider a homogeneous design where the same number of experts of the same size are placed uniformly throughout the network. Furthermore, existing MoE works do not consider computational constraints (e.g., FLOPs, latency) to guide their design. To this end, we develop AutoMoE -- a framework for designing heterogeneous MoE's under computational constraints. AutoMoE leverages Neural Architecture Search (NAS) to obtain efficient sparse MoE sub-transformers with 4x inference speedup (CPU) and FLOPs reduction over manually designed Transformers, with parity in BLEU score over dense Transformer and within 1 BLEU point of MoE SwitchTransformer, on aggregate over benchmark datasets for NMT. Heterogeneous search space with dense and sparsely activated Transformer modules (e.g., how many experts? where to place them? what should be their sizes?) allows for adaptive compute -- where different amounts of computations are used for different tokens in the input. Adaptivity comes naturally from routing decisions which send tokens to experts of different sizes. AutoMoE code, data, and trained models are available at https://aka.ms/AutoMoE.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 35 of its 49 references that are in the map. First posted on arXiv 14 Oct 2022.

Builds on

35 of 49 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers28 May
  2. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  3. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  5. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference24 Sep
  6. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models22 Sep
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  8. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  9. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  10. 2021Pay Attention to MLPscited by 367
  11. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  12. 2018Slimmable Neural Networkscited by 301
15 more
  1. 2022Confident Adaptive Language Modelingcited by 297
  2. 2021Hash Layers For Large Sparse Modelscited by 236
  3. 2022Transformer Quality in Linear Timecited by 189
  4. 2019The Evolved Transformercited by 180
  5. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  6. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  7. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  8. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  9. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 104
  10. 2021NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Searchcited by 55
  11. 2021AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Modelscited by 31
  12. 2021AutoBERT-Zero: Evolving BERT Backbone from Scratchcited by 21
  13. 2021Analyzing and Mitigating Interference in Neural Architecture Searchcited by 20
  14. 2021EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillationcited by 15
  15. 2022AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 3; 2026: 2.

9 papers.

  1. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  2. 2025Advancing Expert Specialization for Better MoEcited by 37
  3. 2023LLM Performance Predictors are good initializers for Architecture Searchcited by 29
  4. 2025Composer: A Search Framework for Hybrid Neural Architecture Designcited by 7
  5. 2024A Time Series is Worth Five Experts: Heterogeneous Mixture of Experts for Traffic Flow Predictioncited by 2
  6. 2025MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 2
  7. 2024Adaptive Inference: Theoretical Limits and Unexplored Opportunitiescited by 0
  8. 2026Mixture-of-Experts in Remote Sensing: A Surveycited by 0
  9. 2026Structure Aware Neural Architecture Search for Mixture of Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity
  2. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
  3. 2025Mixture of Experts in Large Language Models
  4. 2025ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restoration
  5. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection
  6. 2026Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints
  7. 2025THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation
  8. 2025DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs