mlsanityData to 14 Oct 2026

Exploring Length Generalization in Large Language Models

arXiv 2207.04901 ↗First posted 11 Jul 2022cs.CL cs.LGcited by 206

Not on the map. Why?

Cited by 37 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The ability to extrapolate from short problem instances to longer ones is an important form of out-of-distribution generalization in reasoning tasks, and is crucial when learning from datasets where longer problem instances are rare. These include theorem proving, solving quantitative mathematics problems, and reading/summarizing novels. In this paper, we run careful empirical studies exploring the length generalization capabilities of transformer-based language models. We first establish that naively finetuning transformers on length generalization tasks shows significant generalization deficiencies independent of model scale. We then show that combining pretrained large language models' in-context learning abilities with scratchpad prompting (asking the model to output solution steps before producing an answer) results in a dramatic improvement in length generalization. We run careful failure analyses on each of the learning modalities and identify common sources of mistakes that highlight opportunities in equipping language models with the ability to generalize to longer problems.

Key facts. Cited by 206 papers in the map (data through 14 Oct 2026). Builds on 31 of its 33 references that are in the map. First posted on arXiv 11 Jul 2022.

Builds on

31 of 33 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  10. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  11. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  12. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
19 more
  1. 2018Universal Transformerscited by 647
  2. 2020Transformers as Soft Reasoners over Languagecited by 270
  3. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  4. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  5. 2020From Local Structures to Size Generalization in Graph Neural Networkscited by 124
  6. 2015Neural GPUs Learn Algorithmscited by 121
  7. 2018Correcting Length Bias in Neural Machine Translationcited by 120
  8. 2019Unlearn Dataset Bias in Natural Language Inference by Fitting the Residualcited by 119
  9. 2021Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networkscited by 117
  10. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 106
  11. 2020Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networkscited by 72
  12. 2020The EOS Decision and Length Extrapolationcited by 46
  13. 2019LSTM Networks Can Perform Dynamic Countingcited by 45
  14. 2020INT: An Inequality Benchmark for Evaluating Generalization in Theorem Provingcited by 44
  15. 2019Location Attention for Extrapolation to Longer Sequencescited by 33
  16. 2022End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinkingcited by 33
  17. 2020What they do when in doubt: a study of inductive biases in seq2seq learnerscited by 20
  18. 2019Do Multi-hop Readers Dream of Reasoning Chains?cited by 14
  19. 2021Staircase Attention for Recurrent Processing of Sequencescited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 17; 2023: 46; 2024: 56; 2025: 42; 2026: 45.

206 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  3. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  4. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 337
  5. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  6. 2024Titans: Learning to Memorize at Test Timecited by 328
  7. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  8. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  9. 2022Transformers Learn Shortcuts to Automatacited by 268
  10. 2023Challenges and Applications of Large Language Modelscited by 254
  11. 2022Reasoning with Language Model Prompting: A Surveycited by 252
  12. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 199
38 more
  1. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  2. 2024The pitfalls of next-token predictioncited by 160
  3. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  4. 2023Teaching Arithmetic to Small Transformerscited by 123
  5. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 115
  6. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 112
  7. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 105
  8. 2024Chain of Thoughtlessness? An Analysis of CoT in Planningcited by 103
  9. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 102
  10. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 86
  11. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
  12. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examplescited by 84
  13. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 81
  14. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 78
  15. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 75
  16. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  17. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  18. 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 70
  19. 2022LAMBADA: Backward Chaining for Automated Reasoning in Natural Languagecited by 66
  20. 2023Length Generalization in Arithmetic Transformerscited by 66
  21. 2022Parsel: Algorithmic Reasoning with Language Models by Composing Decompositionscited by 64
  22. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  23. 2023Coverage-based Example Selection for In-Context Learningcited by 59
  24. 2024Larimar: Large Language Models with Episodic Memory Controlcited by 53
  25. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  26. 2023Do Transformers Parse while Predicting the Masked Word?cited by 48
  27. 2024Enhance Reasoning for Large Language Models in the Game Werewolfcited by 46
  28. 2024Efficient Large Multi-modal Models via Visual Context Compressioncited by 46
  29. 2024Softmax is not Enough (for Sharp Size Generalisation)cited by 46
  30. 2023Extending Context Window of Large Language Models via Semantic Compressioncited by 45
  31. 2022Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysiscited by 42
  32. 2023ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Modelcited by 39
  33. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 36
  34. 2023Understanding the Capabilities of Large Language Models for Automated Planningcited by 34
  35. 2024Probing the limitations of multimodal language models for chemistry and materials researchcited by 32
  36. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 31
  37. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 31
  38. 2022Towards Better Out-of-Distribution Generalization of Neural Algorithmic Reasoning Taskscited by 30

50 most cited of 206 All 206 (JSON)