mlsanityData to 14 Oct 2026

Transformer Working Memory Enables Regular Language Reasoning and Natural Language Length Extrapolation

arXiv 2305.03796 ↗First posted 5 May 2023cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Unlike recurrent models, conventional wisdom has it that Transformers cannot perfectly model regular languages. Inspired by the notion of working memory, we propose a new Transformer variant named RegularGPT. With its novel combination of Weight-Sharing, Adaptive-Depth, and Sliding-Dilated-Attention, RegularGPT constructs working memory along the depth dimension, thereby enabling efficient and successful modeling of regular languages such as PARITY. We further test RegularGPT on the task of natural language length extrapolation and surprisingly find that it rediscovers the local windowed attention effect deemed necessary in prior work for length extrapolation.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 26 of its 46 references that are in the map. First posted on arXiv 5 May 2023.

Builds on

26 of 46 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2020Longformer: The Long-Document Transformercited by 3,413
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  7. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  9. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  10. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  11. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  12. 2018Universal Transformerscited by 647
14 more
  1. 2014Neural Turing Machinescited by 644
  2. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  3. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 288
  4. 2022Transformers Learn Shortcuts to Automatacited by 268
  5. 2019Depth-Adaptive Transformercited by 236
  6. 2022Neural Networks and the Chomsky Hierarchycited by 217
  7. 2022Exploring Length Generalization in Large Language Modelscited by 206
  8. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 127
  9. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 113
  10. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 86
  11. 2021Adaptive Semiparametric Language Modelscited by 83
  12. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  13. 2020Do Transformers Need Deep Long-Range Memorycited by 24
  14. 2022Characterizing Verbatim Short-Term Memory in Neural Language Modelscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 4; 2025: 3; 2026: 3.

13 papers.

Sort:

By area:

  1. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 199
  2. 2023Guiding Language Model Reasoning with Planning Tokenscited by 58
  3. 2024Understanding Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggregationcited by 22
  4. 2024A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practicescited by 15
  5. 2024A Theory for Length Generalization in Learning to Reasoncited by 11
  6. 2025A Review of Incorporating Psychological Theories in LLMscited by 6
  7. 2023Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalabilitycited by 5
  8. 2024Investigating Recurrent Transformers with Dynamic Haltcited by 5
  9. 2025CARMA: Enhanced Compositionality in LLMs via Advanced Regularisation and Mutual Information Alignmentcited by 3
  10. 2025Rethinking Memorization Measures and their Implications in Large Language Modelscited by 1
  11. 2026Fine-tuning vs. In-context Learning in Large Language Models: A Formal Language Learning Perspectivecited by 1
  12. 2026Length Generalization with Log-Depth Recurrent Unitscited by 1
1 more
  1. 2026Log-Depth Recurrent Language Modelingcited by 0