mlsanityData to 7 Oct 2026

Length Generalization with Log-Depth Recurrent Units

arXiv 2605.26035 ↗First posted 25 May 2026cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Length generalization remains a persistent challenge for neural networks: recurrent models tend to suffer from positional biases, while transformers are constrained by fixed computational depth. Regular languages provide a frequently used testbed for evaluating length generalization, as label prediction can be checked for any sequence length. We propose MLP-LDRU, a type of Log-Depth Recurrent Unit, which captures a class of associativity-biased operators designed to approximate recurrence through parallel reduction. We evaluate MLP-LDRU on 21 regular-language tasks, consisting of standard benchmarks and new prefix languages, where it achieves 100% out-of-distribution accuracy on 18 tasks and at least 99.9% on the remaining 3 when increasing max training length, outperforming comparable recurrent and attention-based models. We further evaluate MLP-LDRU beyond regular languages on ListOps and NLP classification benchmarks, where it performs competitively.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 53 of its 79 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

53 of 79 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  6. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  9. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  10. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  11. 2019On the Convergence of Adam and Beyondcited by 834
  12. 2022Simplified State Space Layers for Sequence Modelingcited by 776
38 more
  1. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  2. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  3. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  4. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 305
  5. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  6. 2022Transformers Learn Shortcuts to Automatacited by 260
  7. 2022Neural Networks and the Chomsky Hierarchycited by 214
  8. 2022Exploring Length Generalization in Large Language Modelscited by 204
  9. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  10. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 128
  11. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 126
  12. 2023What Formal Languages Can Transformers Express? A Surveycited by 120
  13. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  14. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 110
  15. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 109
  16. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 97
  17. 2024Looped Transformers for Length Generalizationcited by 91
  18. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 82
  19. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 77
  20. 2024Why are Sensitive Functions Hard for Transformers?cited by 74
  21. 2020A Formal Hierarchy of RNN Architecturescited by 61
  22. 2024The Expressive Capacity of State Space Models: A Formal Language Perspectivecited by 45
  23. 2024A Formal Framework for Understanding Length Generalization in Transformerscited by 44
  24. 2025Log-Linear Attentioncited by 39
  25. 2024Training Neural Networks as Recognizers of Formal Languagescited by 23
  26. 2024Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structurecited by 22
  27. 2019Ordered Memorycited by 20
  28. 2024Arithmetic Transformers Can Length-Generalize in Both Operand Length and Countcited by 18
  29. 2018On Tree-Based Neural Sentence Modelingcited by 16
  30. 2025Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Modelscited by 16
  31. 2024On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languagescited by 15
  32. 2025Structured Sparse Transition Matrices to Enable State Tracking in State-Space Modelscited by 15
  33. 2022Learnable Commutative Monoids for Graph Neural Networkscited by 13
  34. 2023Transformer Working Memory Enables Regular Language Reasoning and Natural Language Length Extrapolationcited by 13
  35. 2023Advancing Regular Language Reasoning in Linear Recurrent Neural Networkscited by 13
  36. 2025How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Biascited by 10
  37. 2016Neural Tree Indexers for Text Understandingcited by 9
  38. 2025Sequential-Parallel Duality in Prefix Scannable Modelscited by 8

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Log-Depth Recurrent Language Modelingcited by 0