mlsanityData to 7 Oct 2026

Towards Infinite Length Extrapolation: A Unified Approach

arXiv 2601.06113 ↗First posted 3 Jan 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have revolutionized natural language processing, but their ability to process long sequences is fundamentally limited by the context window size during training. Existing length extrapolation methods often suffer from performance degradation or computational inefficiencies. We thereby use a unified framework that reinterprets positional encoding methods as a decomposition of the attention score into a multiplicative transformation and an additive bias. This perspective not only subsumes popular approaches such as relative position embeddings and attention-bias moderated approaches but also exposes their inherent limitations in handling long-range dependencies. To address these shortcomings, motivated by our framework, we introduce Adaptive Positional Encoding (APE), which leverages adaptive frequency modulation and an intricately designed decay bias that incorporates linear, logarithmic, and square-root terms. Our theoretical analysis establishes conditions for infinite-context extrapolation, ensuring that the softmax normalization remains well-defined over unbounded sequences while preserving long-distance correlations, entropy boundedness and gradient positional sensitivity. We substantiate our claims with an experimental case study on TinyStories dataset as well as a new synthetic dataset, Long Tiny Stories featuring stories up to 32,000 words. Relevant code, dataset and model weights are available at https://anonymous.4open.science/r/Check-2DAD/.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 30 references that are in the map. First posted on arXiv 3 Jan 2026.

Builds on

29 of 30 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2020Longformer: The Long-Document Transformercited by 3,362
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  6. 2018Self-Attention with Relative Position Representationscited by 1,112
  7. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  8. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  9. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  10. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  11. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  12. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
17 more
  1. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  2. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  3. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  4. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  5. 2023Retrieval meets Long Context Large Language Modelscited by 114
  6. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  7. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  8. 2023LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuningcited by 47
  9. 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 37
  10. 2023Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encodingcited by 32
  11. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
  12. 2024Are Long-LLMs A Necessity For Long-Context Tasks?cited by 22
  13. 2024HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolationcited by 17
  14. 2024Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspectivecited by 13
  15. 2024Positional Encoding Helps Recurrent Neural Networks Handle a Large Vocabularycited by 5
  16. 2024Training-Free Exponential Context Extension via Cascading KV Cachecited by 4
  17. 2024What Should Baby Models Read? Exploring Sample-Efficient Data Composition on Model Performancecited by 4

Built on by

Not cited yet.