mlsanityData to 14 Oct 2026

The EOS Decision and Length Extrapolation

arXiv 2010.07174 ↗First posted 14 Oct 2020cs.CLcited by 46

Not on the map. Why?

Abstract (arXiv)

Extrapolation to unseen sequence lengths is a challenge for neural generative models of language. In this work, we characterize the effect on length extrapolation of a modeling decision often overlooked: predicting the end of the generative process through the use of a special end-of-sequence (EOS) vocabulary item. We study an oracle setting - forcing models to generate to the correct sequence length at test time - to compare the length-extrapolative behavior of networks trained to predict EOS (+EOS) with networks not trained to (-EOS). We find that -EOS substantially outperforms +EOS, for example extrapolating well to lengths 10 times longer than those seen at training time in a bracket closing task, as well as achieving a 40% improvement over +EOS in the difficult SCAN dataset length generalization task. By comparing the hidden states and dynamics of -EOS and +EOS models, we observe that +EOS models fail to generalize because they (1) unnecessarily stratify their hidden states by their linear position is a sequence (structures we call length manifolds) or (2) get stuck in clusters (which we refer to as length attractors) once the EOS token is the highest-probability prediction.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 12; 2022: 9; 2023: 7; 2024: 7; 2025: 7; 2026: 4.

46 papers.

Sort:

By area:

  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  2. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  3. 2022Transformers Learn Shortcuts to Automatacited by 268
  4. 2022Exploring Length Generalization in Large Language Modelscited by 206
  5. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 139
  6. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  7. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  8. 2023Length Generalization in Arithmetic Transformerscited by 66
  9. 2021Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representationscited by 55
  10. 2022Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsingcited by 50
  11. 2021Improving Compositional Generalization with Latent Structure and Data Augmentationcited by 46
  12. 2023ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Modelcited by 39
34 more
  1. 2021SHAPE: Shifted Absolute Position Embedding for Transformerscited by 37
  2. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 31
  3. 2022On Neural Architecture Inductive Biases for Relational Taskscited by 26
  4. 2023Interpretability Illusions in the Generalization of Simplified Modelscited by 25
  5. 2022Language model acceptability judgements are not always robust to contextcited by 22
  6. 2024Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannotcited by 22
  7. 2021ReaSCAN: Compositional Reasoning in Language Groundingcited by 19
  8. 2023Improving Length-Generalization in Transformers via Task Hintingcited by 13
  9. 2025Extrapolation by Association: Length Generalization Transfer in Transformerscited by 13
  10. 2021Compositional Attention: Disentangling Search and Retrievalcited by 12
  11. 2024Your Context Is Not an Array: Unveiling Random Access Limitations in Transformerscited by 11
  12. 2021Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modelingcited by 10
  13. 2022Benchmarking Long-tail Generalization with Likelihood Splitscited by 9
  14. 2023Monotonic Location Attention for Length Generalizationcited by 9
  15. 2024Large Language Models Synergize with Automated Machine Learningcited by 9
  16. 2025h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learningcited by 9
  17. 2021Compositional Generalization for Natural Language Interfaces to Web APIscited by 7
  18. 2021A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semanticscited by 6
  19. 2021Iterative Decoding for Compositional Generalization in Transformerscited by 6
  20. 2022Compositional Generalisation with Structured Reordering and Fertility Layerscited by 6
  21. 2024Towards Understanding the Relationship between In-context Learning and Compositional Generalizationcited by 5
  22. 2025Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modelingcited by 4
  23. 2026Length Generalization Bounds for Transformerscited by 4
  24. 2022Recursive Decoding: A Situated Cognition Approach to Compositional Generation in Grounded Language Understandingcited by 3
  25. 2024Provable Length Generalization in Sequence Prediction via Spectral Filteringcited by 3
  26. 2025The Imitation Game: Turing Machine Imitator is Length Generalizable Reasonercited by 3
  27. 2023Evaluating Transformer's Ability to Learn Mildly Context-Sensitive Languagescited by 2
  28. 2024Symbolic Autoencoding for Self-Supervised Sequence Learningcited by 2
  29. 2025The Devil is in the EOS: Sequence Training for Detailed Image Captioningcited by 1
  30. 2026Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architecturescited by 1
  31. 2024Enhancing Length Extrapolation in Sequential Models with Pointer-Augmented Neural Memorycited by 0
  32. 2025Learning Model Successorscited by 0
  33. 2026Generalization in LLM Problem Solving: The Case of the Shortest Pathcited by 0
  34. 2026Beam Search, Self-Consistency, and the Limits of Inference-Time Scaling for Grammar-Constrained Text-to-SQL in Small Language Modelscited by 0