mlsanityData to 7 Oct 2026

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

arXiv 2108.12409 ↗First posted 27 Aug 2021cs.CLcited by 1,085

Not on the map. Why?

Cited by 39 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Since the introduction of the transformer model by Vaswani et al. (2017), a fundamental question has yet to be answered: how does a model achieve extrapolation at inference time for sequences that are longer than it saw during training? We first show that extrapolation can be enabled by simply changing the position representation method, though we find that current methods do not allow for efficient extrapolation. We therefore introduce a simpler and more efficient position method, Attention with Linear Biases (ALiBi). ALiBi does not add positional embeddings to word embeddings; instead, it biases query-key attention scores with a penalty that is proportional to their distance. We show that this method trains a 1.3 billion parameter model on input sequences of length 1024 that extrapolates to input sequences of length 2048, achieving the same perplexity as a sinusoidal position embedding model trained on inputs of length 2048 but training 11% faster and using 11% less memory. ALiBi's inductive bias towards recency also leads it to outperform multiple strong position methods on the WikiText-103 benchmark.

Key facts. Cited by 1,085 papers in the map (data through 7 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 27 Aug 2021.

Builds on

34 of 49 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  11. 2018Self-Attention with Relative Position Representationscited by 1,112
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
22 more
  1. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  2. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  3. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  4. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  5. 2016Using the Output Embedding to Improve Language Modelscited by 266
  6. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  7. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  8. 2018A Simple Method for Commonsense Reasoningcited by 223
  9. 2016A Decomposable Attention Model for Natural Language Inferencecited by 214
  10. 2018Music Transformercited by 207
  11. 2018Scaling Neural Machine Translationcited by 204
  12. 2014Recurrent Neural Network Regularizationcited by 189
  13. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  14. 2016Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modelingcited by 87
  15. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  16. 2020Shortformer: Better Language Modeling using Shorter Inputscited by 76
  17. 2019Improving Transformer Models by Reordering their Sublayerscited by 52
  18. 2021CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddingscited by 47
  19. 2020The EOS Decision and Length Extrapolationcited by 46
  20. 2021Towards mental time travel: a hierarchical memory for reinforcement learning agentscited by 38
  21. 2020DA-Transformer: Distance-aware Transformercited by 23
  22. 2021The Case for Translation-Invariant Self-Attention in Transformer-Based Language Modelscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 55; 2023: 171; 2024: 281; 2025: 278; 2026: 292.

1,085 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  8. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  11. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  12. 2023BloombergGPT: A Large Language Model for Financecited by 923
38 more
  1. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  2. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  3. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  4. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  5. 2022Galactica: A Large Language Model for Sciencecited by 736
  6. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  7. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  8. 2024A Survey on Large Language Models for Code Generationcited by 646
  9. 2023A Comprehensive Overview of Large Language Modelscited by 639
  10. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  11. 2023The Falcon Series of Open Language Modelscited by 526
  12. 2024Large Language Models: A Surveycited by 506
  13. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  14. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  15. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  16. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  17. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  18. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  19. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  20. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  21. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  22. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  23. 2023Adapting Language Models to Compress Contextscited by 350
  24. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  25. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  26. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  27. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  28. 2024Massive Activations in Large Language Modelscited by 277
  29. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 275
  30. 2022Transformers Learn Shortcuts to Automatacited by 260
  31. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  32. 2022Can language models learn from explanations in context?cited by 248
  33. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  34. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  35. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  36. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  37. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  38. 2022Neural Networks and the Chomsky Hierarchycited by 214

50 most cited of 1,085 All 1,085 (JSON)