mlsanityData to 7 Oct 2026

Sparse Universal Transformer

arXiv 2310.07096 ↗First posted 11 Oct 2023cs.CL cs.AIcited by 28

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

The Universal Transformer (UT) is a variant of the Transformer that shares parameters across its layers. Empirical evidence shows that UTs have better compositional generalization than Vanilla Transformers (VTs) in formal language tasks. The parameter-sharing also affords it better parameter efficiency than VTs. Despite its many advantages, scaling UT parameters is much more compute and memory intensive than scaling up a VT. This paper proposes the Sparse Universal Transformer (SUT), which leverages Sparse Mixture of Experts (SMoE) and a new stick-breaking-based dynamic halting mechanism to reduce UT's computation complexity while retaining its parameter efficiency and generalization ability. Experiments show that SUT achieves the same performance as strong baseline models while only using half computation and parameters on WMT'14 and strong generalization results on formal language tasks (Logical inference and CFQ). The new halting mechanism also enables around 50% reduction in computation during inference with very little performance decrease on formal language tasks.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 41 of its 47 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

41 of 47 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Mixture of Attention Heads: Selecting Attention Heads Per Token11 Oct
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  8. 2018Universal Transformerscited by 613
  9. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  10. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  11. 2022Confident Adaptive Language Modelingcited by 297
  12. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
27 more
  1. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  2. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  3. 2022Transformers Learn Shortcuts to Automatacited by 260
  4. 2019Depth-Adaptive Transformercited by 227
  5. 2022Neural Networks and the Chomsky Hierarchycited by 214
  6. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  7. 2018Scaling Neural Machine Translationcited by 204
  8. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 137
  9. 2015Neural GPUs Learn Algorithmscited by 118
  10. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  11. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  12. 2020Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architecturescited by 95
  13. 2022Compositional Semantic Parsing with Large Language Modelscited by 95
  14. 2022Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexitycited by 92
  15. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  16. 2019Compositional generalization in a deep seq2seq model by separating syntax and semanticscited by 56
  17. 2020Very Deep Transformers for Neural Machine Translationcited by 50
  18. 2019Compositional Generalization for Primitive Substitutionscited by 48
  19. 2018The Importance of Being Recurrent for Modeling Hierarchical Structurecited by 42
  20. 2021Systematic Generalization with Edge Transformerscited by 38
  21. 2021Sequence-to-Sequence Learning with Latent Neural Grammarscited by 30
  22. 2021Disentangled Sequence to Sequence Learning for Compositional Generalizationcited by 30
  23. 2015Tree-structured composition in neural networks without tree-structured architecturescited by 23
  24. 2020A Mixture of h-1 Heads is Better than h Headscited by 22
  25. 2019Ordered Memorycited by 20
  26. 2022Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learnerscited by 15
  27. 2020Recursive Top-Down Production for Sentence Generation with Latent Treescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 11; 2026: 8.

28 papers.

Sort:

By area:

  1. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  2. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  3. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  4. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  5. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 55
  6. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  7. 2026A Mechanistic Analysis of Looped Reasoning Language Modelscited by 27
  8. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughtscited by 23
  9. 2025Continuous Thought Machinescited by 21
  10. 2025Block-Recurrent Dynamics in Vision Transformerscited by 16
  11. 2024Scattered Mixture-of-Experts Implementationcited by 15
  12. 2024Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Studycited by 13
16 more
  1. 2026Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deservescited by 11
  2. 2026Hyperloop Transformerscited by 11
  3. 2024Conditional computation in neural networks: principles and research trendscited by 10
  4. 2026SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursioncited by 9
  5. 2025MeSH: Memory-as-State-Highways for Recursive Transformerscited by 8
  6. 2025SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thoughtcited by 7
  7. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Modelscited by 7
  8. 2023Adaptive Computation Modules: Granular Conditional Computation For Efficient Inferencecited by 6
  9. 2024Investigating Recurrent Transformers with Dynamic Haltcited by 5
  10. 2025Zero Token-Driven Deep Thinking in LLMs: Unlocking the Full Potential of Existing Parameters via Cyclic Refinementcited by 5
  11. 2026LT2: Linear-Time Looped Transformerscited by 5
  12. 2026Depth-adaptive Inference of Looped Language Models via Continuous Depth Batchingcited by 4
  13. 2025Hyper-SET: Designing Transformers via Hyperspherical Energy Minimizationcited by 2
  14. 2025Knowing When to Quit: Probabilistic Early Exits for Speech Separationcited by 2
  15. 2026Why Depth Matters in Parallelizable Sequence Models: A Lie Algebraic Viewcited by 1
  16. 2026Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Ultra-Sparse Memory Network
  2. 2026Fixed Universal Transformers
  3. 2025Universal Reasoning Model
  4. 2024Do Efficient Transformers Really Save Computation?
  5. 2025Exact Expressive Power of Transformers with Padding
  6. 2021Taming Sparsely Activated Transformer with Stochastic Experts
  7. 2025LLM Inference Acceleration via Efficient Operation Fusion
  8. 2023Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity