mlsanityData to 14 Oct 2026

Monotonic Location Attention for Length Generalization

arXiv 2305.20019 ↗First posted 31 May 2023cs.LG cs.AI cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

We explore different ways to utilize position-based cross-attention in seq2seq networks to enable length generalization in algorithmic tasks. We show that a simple approach of interpolating the original and reversed encoded representations combined with relative attention allows near-perfect length generalization for both forward and reverse lookup tasks or copy tasks that had been generally hard to tackle. We also devise harder diagnostic tasks where the relative distance of the ideal attention position varies with timestep. In such settings, the simple interpolation trick with relative attention is not sufficient. We introduce novel variants of location attention building on top of Dubois et al. (2020) to address the new diagnostic tasks. We also show the benefits of our approaches for length generalization in SCAN (Lake & Baroni, 2018) and CFQ (Keysers et al., 2020). Our code is available on GitHub.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 38 of its 51 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

38 of 51 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  7. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  9. 2018Self-Attention with Relative Position Representationscited by 1,120
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  11. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  12. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
26 more
  1. 2018Universal Transformerscited by 647
  2. 2014Neural Turing Machinescited by 644
  3. 2019Deep Equilibrium Modelscited by 615
  4. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
  5. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 284
  6. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  7. 2022Exploring Length Generalization in Large Language Modelscited by 206
  8. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  9. 2021PonderNet: Learning to Pondercited by 168
  10. 2022A Length-Extrapolatable Transformercited by 127
  11. 2015Neural GPUs Learn Algorithmscited by 121
  12. 2021Position Information in Transformers: An Overviewcited by 103
  13. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  14. 2022Your Transformer May Not be as Powerful as You Expectcited by 55
  15. 2019Encoding word order in complex embeddingscited by 53
  16. 2021Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encodingcited by 51
  17. 2021CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddingscited by 47
  18. 2020The EOS Decision and Length Extrapolationcited by 46
  19. 2018Modeling Localness for Self-Attention Networkscited by 43
  20. 2018Memorize or generalize? Searching for a compositional RNN in a haystackcited by 41
  21. 2019Location Attention for Extrapolation to Longer Sequencescited by 33
  22. 2022LISA: Learning Interpretable Skill Abstractions from Languagecited by 30
  23. 2022Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Modelscited by 26
  24. 2020DA-Transformer: Distance-aware Transformercited by 23
  25. 2021The Case for Translation-Invariant Self-Attention in Transformer-Based Language Modelscited by 23
  26. 2021Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Modelscited by 11

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 5; 2025: 2.

9 papers.

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  2. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 62
  3. 2024Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolationcited by 24
  4. 2023Functional Interpolation for Relative Positions Improves Long Context Transformerscited by 18
  5. 2024A Theory for Length Generalization in Learning to Reasoncited by 11
  6. 2025HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Modelscited by 10
  7. 2024Equipping Transformer with Random-Access Reading for Long-Context Understandingcited by 1
  8. 2024Out-of-distribution generalisation in spoken language understandingcited by 1
  9. 2025TRA: Better Length Generalisation with Threshold Relative Attentioncited by 1