mlsanityData to 14 Oct 2026

Temporal Recurrence Favors Fewer Layers

arXiv 2609.12531 ↗First posted 11 Sep 2026cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

In streaming tasks, recurrent models can carry latent computation across time, allowing each update to build on representations produced earlier. This raises a basic question: once temporal recurrence provides sequential computation across steps, how much depth is still needed within each step? Prior work has shown that recurrence can make shallow models competitive. We instead study this question as a compute-allocation problem, varying within-step depth, expert width, and the number of parallel experts per layer across several compute budgets. For each budget, we compare the best observed recurrent and non-recurrent allocations and the performance they achieve under approximately matched per-step computation. Across Sokoban and autoregressive FineWeb language modeling, we find that temporal recurrence shifts the best observed compute allocation toward substantially fewer layers, with comparable or better performance.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 41 of its 59 references that are in the map. First posted on arXiv 11 Sep 2026.

Builds on

41 of 59 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  12. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
29 more
  1. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  2. 2018Universal Transformerscited by 647
  3. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
  4. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  5. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  6. 2016Learning to reinforcement learncited by 377
  7. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 320
  8. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 301
  9. 2023From Sparse to Soft Mixtures of Expertscited by 248
  10. 2019Recurrent Independent Mechanismscited by 219
  11. 2024The Illusion of State in State-Space Modelscited by 187
  12. 2021PonderNet: Learning to Pondercited by 168
  13. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  14. 2015Memory-based control with recurrent neural networkscited by 78
  15. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 78
  16. 2019An investigation of model-free planningcited by 66
  17. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 59
  18. 2024Transformer Layers as Painterscited by 54
  19. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 45
  20. 2016Recurrent Highway Networkscited by 37
  21. 2025Continuous Thought Machinescited by 22
  22. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  23. 2024Planning in a recurrent neural network that plays Sokobancited by 18
  24. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RLcited by 14
  25. 2023Thinker: Learning to Plan and Actcited by 13
  26. 2026Why Are Linear RNNs More Parallelizable?cited by 8
  27. 2018Recurrent Predictive State Policy Networkscited by 6
  28. 2026The Recurrent Transformer: Greater Effective Depth and Efficient Decodingcited by 6
  29. 2024LayerShuffle: Enhancing Robustness in Vision Transformers by Randomizing Layer Execution Ordercited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Looped Actor: Depth-Recurrent Reasoning Models for Reinforcement Learningcited by 0