mlsanityData to 7 Oct 2026

Efficient Language Modeling with Sparse all-MLP

arXiv 2203.06850 ↗First posted 14 Mar 2022cs.CL cs.AIcited by 8

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

All-MLP architectures have attracted increasing interest as an alternative to attention-based models. In NLP, recent work like gMLP shows that all-MLPs can match Transformers in language modeling, but still lag behind in downstream tasks. In this work, we analyze the limitations of MLPs in expressiveness, and propose sparsely activated MLPs with mixture-of-experts (MoEs) in both feature and input (token) dimensions. Such sparse all-MLPs significantly increase model capacity and expressiveness while keeping the compute constant. We address critical challenges in incorporating conditional computation with two routing strategies. The proposed sparse all-MLP improves language modeling perplexity and obtains up to 2× improvement in training efficiency compared to both Transformer-based MoEs (GShard, Switch Transformer, Base Layers and HASH Layers) as well as dense Transformers and all-MLPs. Finally, we evaluate its zero-shot in-context learning performance on six downstream tasks, and find that it surpasses Transformer-based MoEs and dense Transformers.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 25 of its 40 references that are in the map. First posted on arXiv 14 Mar 2022.

Builds on

25 of 40 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  10. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  11. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  12. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
10 more
  1. 2021Pay Attention to MLPscited by 367
  2. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  3. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  4. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  5. 2021Hash Layers For Large Sparse Modelscited by 236
  6. 2018A Simple Method for Commonsense Reasoningcited by 223
  7. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  8. 2021Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognitioncited by 103
  9. 2021Sparse is Enough in Scaling Transformerscited by 86
  10. 2021M6-T: Exploring Sparse Expert Models and Beyondcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 1; 2026: 1.

8 papers.

  1. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  2. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  3. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  4. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 56
  5. 2024Setting the Record Straight on Transformer Oversmoothingcited by 19
  6. 2025MLP Memory: A Retriever-Pretrained Memory for Large Language Modelscited by 19
  7. 2024Turn Waste into Worth: Rectifying Top-k Router of MoEcited by 11
  8. 2026RAFNet: Region-Aware Fusion Network for Pansharpeningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
  2. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models
  3. 2023Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity
  4. 2023Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model
  5. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
  6. 2023Adaptive Gating in Mixture-of-Experts based Language Models
  7. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  8. 2025MLPMoE: Zero-Shot Architectural Metamorphosis of Dense LLM MLPs into Static Mixture-of-Experts