mlsanityData to 14 Oct 2026

Enhancing Length Extrapolation in Sequential Models with Pointer-Augmented Neural Memory

arXiv 2404.11870 ↗First posted 18 Apr 2024cs.LG cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We propose Pointer-Augmented Neural Memory (PANM) to help neural networks understand and apply symbol processing to new, longer sequences of data. PANM integrates an external neural memory that uses novel physical addresses and pointer manipulation techniques to mimic human and computer symbol processing abilities. PANM facilitates pointer assignment, dereference, and arithmetic by explicitly using physical pointers to access memory content. Remarkably, it can learn to perform these operations through end-to-end training on sequence data, powering various sequential models. Our experiments demonstrate PANM's exceptional length extrapolating capabilities and improved performance in tasks that require symbol processing, such as algorithmic reasoning and Dyck language recognition. PANM helps Transformer achieve up to 100% generalization accuracy in compositional learning tasks and significantly better results in mathematical reasoning, question answering and machine translation tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 41 of its 49 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

41 of 49 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  7. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  8. 2015Pointer Networkscited by 895
  9. 2018Universal Transformerscited by 647
  10. 2014Neural Turing Machinescited by 644
  11. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
  12. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
29 more
  1. 2022Neural Networks and the Chomsky Hierarchycited by 217
  2. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 165
  3. 2019Good-Enough Compositional Data Augmentationcited by 158
  4. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
  5. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  6. 2019Compositional generalization through meta sequence-to-sequence learningcited by 99
  7. 2020Compositional Generalization via Neural-Symbolic Stack Machinescited by 80
  8. 2020Learning Compositional Rules via Neural Program Synthesiscited by 74
  9. 2022Limitations of Language Models in Arithmetic and Symbolic Inductioncited by 71
  10. 2023Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasonerscited by 63
  11. 2019Improving Deep Transformer with Depth-Scaled Initialization and Merged Attentioncited by 59
  12. 2019Compositional generalization in a deep seq2seq model by separating syntax and semanticscited by 56
  13. 2015Learning to Transduce with Unbounded Memorycited by 54
  14. 2020Compositional Generalization by Learning Analytical Expressionscited by 53
  15. 2020Emergent Symbols through Binding in External Memorycited by 47
  16. 2020The EOS Decision and Length Extrapolationcited by 46
  17. 2015Neural Random-Access Machinescited by 34
  18. 2019Location Attention for Extrapolation to Longer Sequencescited by 33
  19. 2020Self-Attentive Associative Memorycited by 33
  20. 2019Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languagescited by 31
  21. 2018Dual Memory Neural Computer for Asynchronous Two-view Sequential Learningcited by 27
  22. 2022Systematic Generalization and Emergent Structures in Transformers Trained on Structured Taskscited by 19
  23. 2020Systematic Generalization on gSCAN with Language Conditioned Embeddingcited by 15
  24. 2018Context-Free Transductions with Neural Stackscited by 14
  25. 2019Neural Stored-program Memorycited by 14
  26. 2018DeepProcess: Supporting business process execution using a MANN-based recommender systemcited by 6
  27. 2021Generalization in Multimodal Language Learning from Simulationcited by 6
  28. 2020Analogical Reasoning for Visually Grounded Language Acquisitioncited by 4
  29. 2016Lie Access Neural Turing Machinecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Position as Probability: Self-Supervised Transformers that Think Past Their Training for Length Extrapolation
  2. 2022Neural-Symbolic Recursive Machine for Systematic Generalization
  3. 2025To Infinity and Beyond: Tool-Use Unlocks Length Generalization in State Space Models
  4. 2024Explicitly Encoding Structural Symmetry is Key to Length Generalization in Arithmetic Tasks
  5. 2024Universal Length Generalization with Turing Programs
  6. 2024Memory3: Language Modeling with Explicit Memory
  7. 2025Sequential Enumeration in Large Language Models
  8. 2023A Hybrid System for Systematic Generalization in Simple Arithmetic Problems