Context-aware Biases for Length Extrapolation
Not on the map. Why?
Abstract (arXiv)
Transformers often struggle to generalize to longer sequences than those seen during training, a limitation known as length extrapolation. Most existing Relative Positional Encoding (RPE) methods attempt to address this by introducing either fixed linear biases or globally learned biases, which lack the capacity to adapt to different input contexts. In this work, we propose an additive RPE, Context-Aware Biases for Length Extrapolation (CABLE), a method that learns token-specific, context-aware biases for each attention head in transformers. By dynamically adjusting positional biases based on the input sequence, CABLE overcomes the rigidity of fixed RPEs. When evaluated on sequences longer than originally trained with, GPT-2 Medium (334M parameters) with CABLE achieves lower perplexity than counterparts using other widely adopted positional encoding methods. Additionally, by applying CABLE to the BERT base model we improved performance in long-context retrieval tasks. Our method significantly enhances the extrapolation performance of existing RPE methods tested on the FineWeb-Edu-10B and WikiText-103 datasets. Our code is available at: https://github.com/AlgonetLabs/Cable.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 59 of its 67 references that are in the map. First posted on arXiv 11 Mar 2025.
Builds on
59 of 67 references.
References · 59
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023GPT-4 Technical Reportcited by 18,132
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2020Longformer: The Long-Document Transformercited by 3,362
38 more
- 2016Pointer Sentinel Mixture Modelscited by 2,992
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
- 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
- 2018Self-Attention with Relative Position Representationscited by 1,112
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
- 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
- 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
- 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
- 2017Convolutional Sequence to Sequence Learningcited by 550
- 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
- 2023The Falcon Series of Open Language Modelscited by 526
- 2018Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Networkcited by 415
- 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
- 2023Learning to Compress Prompts with Gist Tokenscited by 379
- 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
- 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
- 2018Adaptive Input Representations for Neural Language Modelingcited by 226
- 2018Scaling Neural Machine Translationcited by 204
- 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
- 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 148
- 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
- 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
- 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
- 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
- 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
- 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
- 2020Hard negative examples are hard, but usefulcited by 82
- 2023Positional Description Matters for Transformers Arithmeticcited by 56
- 2024Contextual Position Encoding: Learning to Count What's Importantcited by 53
- 2024Base of RoPE Bounds Context Lengthcited by 49
- 2021CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddingscited by 47
- 2024Length Generalization of Causal Transformers without Position Encodingcited by 46
- 2022Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysiscited by 39
- 2021A Simple and Effective Positional Encoding for Transformerscited by 37
- 2021SHAPE: Shifted Absolute Position Embedding for Transformerscited by 37
50 most cited of 59 All 59 (JSON)
Built on by
Citing papers per year: 2025: 2.
2 papers.