mlsanityData to 7 Oct 2026

Investigating Recurrent Transformers with Dynamic Halt

arXiv 2402.00976 ↗First posted 1 Feb 2024cs.LG cs.AI cs.NEcited by 5

Not on the map. Why?

Abstract (arXiv)

In this paper, we comprehensively study the inductive biases of two major approaches to augmenting Transformers with a recurrent mechanism: (1) the approach of incorporating a depth-wise recurrence similar to Universal Transformers; and (2) the approach of incorporating a chunk-wise temporal recurrence like Temporal Latent Bottleneck. Furthermore, we propose and investigate novel ways to extend and combine the above methods - for example, we propose a global mean-based dynamic halting mechanism for Universal Transformers and an augmentation of Temporal Latent Bottleneck with elements from Universal Transformer. We compare the models and probe their inductive biases in several diagnostic tasks, such as Long Range Arena (LRA), flip-flop language modeling, ListOps, and Logical Inference. The code is released in: https://github.com/JRC1995/InvestigatingRecurrentTransformers/tree/main

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 80 of its 106 references that are in the map. First posted on arXiv 1 Feb 2024.

Builds on

80 of 106 references.

References · 80

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2016Layer Normalizationcited by 5,579
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2020GLU Variants Improve Transformercited by 1,724
  11. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  12. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
38 more
  1. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  2. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  3. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  4. 2016Language Modeling with Gated Convolutional Networkscited by 688
  5. 2021Transformer in Transformercited by 632
  6. 2014Neural Turing Machinescited by 630
  7. 2018Universal Transformerscited by 613
  8. 2019Deep Equilibrium Modelscited by 603
  9. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  10. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  11. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  12. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  13. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  14. 2021Dynamic Neural Networks: A Surveycited by 405
  15. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  16. 2022Confident Adaptive Language Modelingcited by 297
  17. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  18. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  19. 2022Long Range Language Modeling via Gated State Spacescited by 265
  20. 2022Transformers Learn Shortcuts to Automatacited by 260
  21. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  22. 2019Depth-Adaptive Transformercited by 227
  23. 2022Neural Networks and the Chomsky Hierarchycited by 214
  24. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  25. 2022Recurrent Memory Transformercited by 197
  26. 2022The Parallelism Tradeoff: Limitations of Log-Precision Transformerscited by 191
  27. 2022Transformer Quality in Linear Timecited by 189
  28. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  29. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 137
  30. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 128
  31. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 126
  32. 2022A Length-Extrapolatable Transformercited by 123
  33. 2015Neural GPUs Learn Algorithmscited by 118
  34. 2022Block-Recurrent Transformerscited by 107
  35. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  36. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  37. 2022Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexitycited by 92
  38. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025A Survey on Latent Reasoningcited by 71
  2. 2026Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoningcited by 12
  3. 2026HRM-Text: Efficient Pretraining Beyond Scalingcited by 7
  4. 2026Loop the Loopies!cited by 4
  5. 2025Bridging Reasoning to Learning: Unmasking Illusions using Complexity Out of Distribution Generalizationcited by 1