mlsanityData to 14 Oct 2026

Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure

arXiv 2405.20671 ↗First posted 31 May 2024cs.LG cs.AI cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

Even for simple arithmetic tasks like integer addition, it is challenging for Transformers to generalize to longer sequences than those encountered during training. To tackle this problem, we propose position coupling, a simple yet effective method that directly embeds the structure of the tasks into the positional encoding of a (decoder-only) Transformer. Taking a departure from the vanilla absolute position mechanism assigning unique position IDs to each of the tokens, we assign the same position IDs to two or more "relevant" tokens; for integer addition tasks, we regard digits of the same significance as in the same position. On the empirical side, we show that with the proposed position coupling, our models trained on 1 to 30-digit additions can generalize up to 200-digit additions (6.67x of the trained length). On the theoretical side, we prove that a 1-layer Transformer with coupled positions can solve the addition task involving exponentially many digits, whereas any 1-layer Transformer without positional information cannot entirely solve it. We also demonstrate that position coupling can be applied to other algorithmic tasks such as Nx2 multiplication and a two-dimensional task.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 42 of its 50 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

42 of 50 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2016Layer Normalizationcited by 5,669
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  9. 2020GLU Variants Improve Transformercited by 1,804
  10. 2019Root Mean Square Layer Normalizationcited by 1,480
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  12. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
30 more
  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  2. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  3. 2016Language Modeling with Gated Convolutional Networkscited by 702
  4. 2017Convolutional Sequence to Sequence Learningcited by 552
  5. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  6. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  7. 2022Autoformalization with Large Language Modelscited by 263
  8. 2022Neural Networks and the Chomsky Hierarchycited by 217
  9. 2022Exploring Length Generalization in Large Language Modelscited by 206
  10. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 199
  11. 2021Thinking Like Transformerscited by 184
  12. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 139
  13. 2023Teaching Arithmetic to Small Transformerscited by 123
  14. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 117
  15. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 106
  16. 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 104
  17. 2020Low-Rank Bottleneck in Multi-head Attention Modelscited by 89
  18. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 86
  19. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 78
  20. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  21. 2023Length Generalization in Arithmetic Transformerscited by 66
  22. 2023Positional Description Matters for Transformers Arithmeticcited by 56
  23. 2023Learning Transformer Programscited by 41
  24. 2023Understanding Addition in Transformerscited by 34
  25. 2024On Provable Length and Compositional Generalizationcited by 22
  26. 2023Functional Interpolation for Relative Positions Improves Long Context Transformerscited by 18
  27. 2023Improving Length-Generalization in Transformers via Task Hintingcited by 13
  28. 2024A Theory for Length Generalization in Learning to Reasoncited by 11
  29. 2023From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformerscited by 10
  30. 2023Conditions for Length Generalization in Learning Reasoning Skillscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 10.

23 papers.

Sort:

By area:

  1. 2025Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lenscited by 43
  2. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 31
  3. 2025Extrapolation by Association: Length Generalization Transfer in Transformerscited by 13
  4. 2025Beyond Single-Task: Robust Multi-Task Length Generalization for LLMscited by 7
  5. 2025Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilitiescited by 6
  6. 2025A Theoretical Framework for OOD Robustness in Transformers using Gevrey Classescited by 4
  7. 2025To Infinity and Beyond: Tool-Use Unlocks Length Generalization in State Space Modelscited by 4
  8. 2026Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stoppingcited by 4
  9. 2025Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More Morecited by 3
  10. 2026On the Role of Computation in Reinforcement Learningcited by 3
  11. 2026On the Ability of Transformers to Verify Planscited by 2
  12. 2025Exact Learning of Arithmetic with Differentiable Agentscited by 1
11 more
  1. 2025Pre-Generating Multi-Difficulty PDE Data for Few-Shot Neural PDE Solverscited by 1
  2. 2025From Small to Large: Generalization Bounds for Transformers on Variable-Size Inputscited by 1
  3. 2026Arithmetic OOD Failure Unfolds in Stages in Minimal GPTscited by 1
  4. 2026Training Transformers as a Universal Computercited by 1
  5. 2026Length Generalization with Log-Depth Recurrent Unitscited by 1
  6. 2025Position as Probability: Self-Supervised Transformers that Think Past Their Training for Length Extrapolationcited by 0
  7. 2025On the Limitations and Capabilities of Position Embeddings for Length Generalizationcited by 0
  8. 2026Convex Optimization of Spiking Neural Networks with Global and Approximation Guaranteescited by 0
  9. 2026Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attentioncited by 0
  10. 2026Beyond the Training Horizon: Mechanisms and Limits of Length Generalization in Looped Transformerscited by 0
  11. 2026Counting with Attention and LayerNorm: Norm Scaling and Representation Geometrycited by 0