mlsanityData to 14 Oct 2026

Universal Transformers

arXiv 1807.03819 ↗First posted 10 Jul 2018cs.CL cs.LG stat.MLcited by 647

Not on the map. Why?

Cited by 98 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recurrent neural networks (RNNs) sequentially process data by updating their state with each new data point, and have long been the de facto choice for sequence modeling tasks. However, their inherently sequential computation makes them slow to train. Feed-forward and convolutional architectures have recently been shown to achieve superior results on some sequence modeling tasks such as machine translation, with the added advantage that they concurrently process all inputs in the sequence, leading to easy parallelization and faster training times. Despite these successes, however, popular feed-forward sequence models like the Transformer fail to generalize in many simple tasks that recurrent models handle with ease, e.g. copying strings or even simple logical inference when the string or formula lengths exceed those observed at training time. We propose the Universal Transformer (UT), a parallel-in-time self-attentive recurrent sequence model which can be cast as a generalization of the Transformer model and which addresses these issues. UTs combine the parallelizability and global receptive field of feed-forward sequence models like the Transformer with the recurrent inductive bias of RNNs. We also add a dynamic per-position halting mechanism and find that it improves accuracy on several tasks. In contrast to the standard Transformer, under certain assumptions, UTs can be shown to be Turing-complete. Our experiments show that UTs outperform standard Transformers on a wide range of algorithmic and language understanding tasks, including the challenging LAMBADA language modeling task where UTs achieve a new state of the art, and machine translation where UTs achieve a 0.9 BLEU improvement over Transformers on the WMT14 En-De dataset.

Key facts. Cited by 647 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Jul 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 99; 2022: 78; 2023: 80; 2024: 61; 2025: 84; 2026: 245.

647 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Emergent Abilities of Large Language Modelscited by 2,512
  3. 2021ViViT: A Video Vision Transformercited by 1,620
  4. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  5. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  6. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  7. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  8. 2023LightGlue: Local Feature Matching at Light Speedcited by 590
  9. 2022Transformers in Time Series: A Surveycited by 497
  10. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  11. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  12. 2021Dynamic Neural Networks: A Surveycited by 408
38 more
  1. 2023Scaling Data-Constrained Language Modelscited by 403
  2. 2021A Survey of Transformerscited by 401
  3. 2021Random Feature Attentioncited by 354
  4. 2021Choose a Transformer: Fourier or Galerkincited by 343
  5. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  6. 2022Confident Adaptive Language Modelingcited by 310
  7. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  8. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 294
  9. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 271
  10. 2022Transformers Learn Shortcuts to Automatacited by 268
  11. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  12. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 240
  13. 2023Looped Transformers as Programmable Computerscited by 237
  14. 2022Exploring Length Generalization in Large Language Modelscited by 206
  15. 2022The Parallelism Tradeoff: Limitations of Log-Precision Transformerscited by 193
  16. 2021Thinking Like Transformerscited by 184
  17. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  18. 2025Scaling Latent Reasoning via Looped Language Modelscited by 165
  19. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 146
  20. 2021Deep Learning Through the Lens of Example Difficultycited by 141
  21. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141
  22. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 139
  23. 2021The Benchmark Lotterycited by 136
  24. 2022Language Models as Agent Modelscited by 132
  25. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  26. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  27. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 124
  28. 2022RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Datacited by 123
  29. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  30. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 117
  31. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 109
  32. 2021Self-Attention Networks Can Process Bounded Hierarchical Languagescited by 105
  33. 2021Position Information in Transformers: An Overviewcited by 103
  34. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  35. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 102
  36. 2021Finetuning Pretrained Transformers into RNNscited by 96
  37. 2021Luna: Linear Unified Nested Attentioncited by 96
  38. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 92

50 most cited of 647 All 647 (JSON)