mlsanityData to 7 Oct 2026

MoEUT: Mixture-of-Experts Universal Transformers

arXiv 2405.16039 ↗First posted 25 May 2024cs.LG cs.AI cs.NEcited by 55

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Previous work on Universal Transformers (UTs) has demonstrated the importance of parameter sharing across layers. By allowing recurrence in depth, UTs have advantages over standard Transformers in learning compositional generalizations, but layer-sharing comes with a practical limitation of parameter-compute ratio: it drastically reduces the parameter count compared to the non-shared model with the same dimensionality. Naively scaling up the layer size to compensate for the loss of parameters makes its computational resource requirements prohibitive. In practice, no previous work has succeeded in proposing a shared-layer Transformer design that is competitive in parameter count-dominated tasks such as language modeling. Here we propose MoEUT (pronounced "moot"), an effective mixture-of-experts (MoE)-based shared-layer Transformer architecture, which combines several recent advances in MoEs for both feedforward and attention layers of standard Transformers together with novel layer-normalization and grouping schemes that are specific and crucial to UTs. The resulting UT model, for the first time, slightly outperforms standard Transformers on language modeling tasks such as BLiMP and PIQA, while using significantly less compute and memory.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 46 of its 61 references that are in the map. First posted on arXiv 25 May 2024.

Builds on

46 of 61 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention13 Dec
  2. 2023Sparse Universal Transformer11 Oct
  3. 2022Mixture of Attention Heads: Selecting Attention Heads Per Token11 Oct
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2016Layer Normalizationcited by 5,579
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
30 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  10. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  11. 2022In-context Learning and Induction Headscited by 930
  12. 2022Toy Models of Superpositioncited by 928
  13. 2020On Layer Normalization in the Transformer Architecturecited by 906
  14. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  15. 2018Universal Transformerscited by 613
  16. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  17. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  18. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  19. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  20. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  21. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  22. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  23. 2015The Goldilocks Principle: Reading Children's Books with Explicit Memory Representationscited by 108
  24. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  25. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  26. 2021Making Transformers Solve Compositional Taskscited by 72
  27. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  28. 2021Systematic Generalization with Edge Transformerscited by 38
  29. 2023ResiDual: Transformer with Dual Residual Connectionscited by 30
  30. 2023Approximating Two-Layer Feedforward Networks for Efficient Transformerscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 23; 2026: 28.

55 papers.

Sort:

By area:

  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  2. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  3. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  4. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 48
  5. 2025Do Language Models Use Their Depth Efficiently?cited by 48
  6. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  7. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 45
  8. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  9. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughtscited by 23
  10. 2025Improved Representation Steering for Language Modelscited by 17
  11. 2025A Formal Comparison Between Chain of Thought and Latent Thoughtcited by 14
  12. 2026Training-Free Looped Transformerscited by 12
38 more
  1. 2026SMELT: Scaling Laws for Compute-Matched MoE Looped Transformerscited by 12
  2. 2026Hyperloop Transformerscited by 11
  3. 2025To CoT or To Loop? A Formal Comparison Between Chain-of-Thought and Looped Transformerscited by 10
  4. 2025From Markov to Laplace: How Mamba In-Context Learns Markov Chainscited by 9
  5. 2025Learning to Skip the Middle Layers of Transformerscited by 8
  6. 2025MeSH: Memory-as-State-Highways for Recursive Transformerscited by 8
  7. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Modelscited by 7
  8. 2025Efficient Parallel Samplers for Recurrent-Depth Models and Their Connection to Diffusion Language Modelscited by 7
  9. 2025ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Expertscited by 7
  10. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  11. 2025UMoE: Unifying Attention and FFN with Shared Expertscited by 6
  12. 2026From Growing to Looping: A Unified View of Iterative Computation in LLMscited by 6
  13. 2025Zero Token-Driven Deep Thinking in LLMs: Unlocking the Full Potential of Existing Parameters via Cyclic Refinementcited by 5
  14. 2025Do Depth-Grown Models Overcome the Curse of Depth? An In-Depth Analysiscited by 5
  15. 2026LT2: Linear-Time Looped Transformerscited by 5
  16. 2025Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognitioncited by 4
  17. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  18. 2026Looping Back to Move Forward: Recursive Transformers for Efficient and Flexible Large Multimodal Modelscited by 4
  19. 2026Loop the Loopies!cited by 4
  20. 2026Allocating Recurrent Compute in Looped Language Modelscited by 4
  21. 2025Hyper-SET: Designing Transformers via Hyperspherical Energy Minimizationcited by 2
  22. 2025Mixture of States: Routing Token-Level Dynamics for Multimodal Generationcited by 2
  23. 2026Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Modelscited by 2
  24. 2026Looped World Modelscited by 2
  25. 2026Why Depth Matters in Parallelizable Sequence Models: A Lie Algebraic Viewcited by 1
  26. 2026UniPool: A Globally Shared Expert Pool for Mixture-of-Expertscited by 1
  27. 2026Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Modelscited by 1
  28. 2026LiMoDE: Rethinking Lifelong Robot Manipulation from a Mixture-of-Dynamic-Experts Perspectivecited by 1
  29. 2026Multi-level context Modeling for consistent expert selection in Mixture-of-Expertscited by 1
  30. 2026Temporal Recurrence Favors Fewer Layerscited by 1
  31. 2026MoRE: Mixture of Reused Expertscited by 1
  32. 2026Programs-of-Layers in LLMs through the Lens of Cortical Areascited by 1
  33. 2025Optimal Scaling Laws for Efficiency Gains in a Theoretical Transformer-Augmented Sectional MoE Frameworkcited by 0
  34. 2025ShishuLM : Achieving Optimal and Efficient Parameterization with Low Attention Transformer Modelscited by 0
  35. 2026LA-Sign: Looped Transformers with Geometry-aware Alignment for Skeleton-based Sign Language Recognitioncited by 0
  36. 2026Mixture of Layers with Hybrid Attentioncited by 0
  37. 2026Scaling Adaptive Depth with Norm-Agnostic Residual Networkscited by 0
  38. 2026Repeated Shared Access Enables Grokking, but Edit Propagation Depends on an Addressable Memorycited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Mixture of Tokens: Continuous MoE through Cross-Example Aggregation
  2. 2026Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers
  3. 2026A theoretical model for task routing in mixture-of-expert transformers
  4. 2021Efficient Large Scale Language Modeling with Mixtures of Experts
  5. 2023Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity
  6. 2024Mixture of A Million Experts
  7. 2024A Survey on Mixture of Experts in Large Language Models
  8. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models