mlsanityData to 7 Oct 2026

Extending Context Window of Large Language Models via Positional Interpolation

arXiv 2306.15595 ↗First posted 27 Jun 2023cs.CL cs.AI cs.LGcited by 721

Not on the map. Why?

Cited by 64 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps), while demonstrating strong empirical results on various tasks that require long context, including passkey retrieval, language modeling, and long document summarization from LLaMA 7B to 65B. Meanwhile, the extended model by Position Interpolation preserve quality relatively well on tasks within its original context window. To achieve this goal, Position Interpolation linearly down-scales the input position indices to match the original context window size, rather than extrapolating beyond the trained context length which may lead to catastrophically high attention scores that completely ruin the self-attention mechanism. Our theoretical study shows that the upper bound of interpolation is at least ~ 600 × smaller than that of extrapolation, further demonstrating its stability. Models extended via Position Interpolation retain its original architecture and can reuse most pre-existing optimization and infrastructure.

Key facts. Cited by 721 papers in the map (data through 7 Oct 2026). Builds on 36 of its 47 references that are in the map. First posted on arXiv 27 Jun 2023.

Builds on

36 of 47 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2020Longformer: The Long-Document Transformercited by 3,362
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
24 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  4. 2020Reformer: The Efficient Transformercited by 1,658
  5. 2020Rethinking Attention with Performerscited by 1,584
  6. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  9. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  10. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  11. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  12. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  13. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  14. 2021Efficient Attentions for Long Document Summarizationcited by 342
  15. 2022Memorizing Transformerscited by 219
  16. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  17. 2022Recurrent Memory Transformercited by 197
  18. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  19. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 148
  20. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  21. 2020Relevance-guided Supervision for OpenQA with ColBERTcited by 71
  22. 2021Combiner: Full Attention Transformer with Sparse Computation Costcited by 64
  23. 2020Memformer: A Memory-Augmented Transformer for Sequence Modelingcited by 63
  24. 2022Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformercited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 89; 2024: 265; 2025: 201; 2026: 166.

721 papers.

Sort:

By area:

  1. 2023Qwen Technical Reportcited by 3,273
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  5. 2025Gemma 3 Technical Reportcited by 1,686
  6. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  8. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  10. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  11. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  12. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
38 more
  1. 2023A Comprehensive Overview of Large Language Modelscited by 639
  2. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  4. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  5. 2023The Falcon Series of Open Language Modelscited by 526
  6. 2024Long Context Transfer from Language to Visioncited by 502
  7. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  8. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 401
  9. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  10. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  11. 2025Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Passcited by 356
  12. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  13. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  14. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  15. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  16. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  17. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  18. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  19. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  20. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  21. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  22. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  23. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  24. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  25. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  26. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  27. 2023Efficient Large Language Models: A Surveycited by 183
  28. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  29. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  30. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 176
  31. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  32. 2025Phi-4-reasoning Technical Reportcited by 146
  33. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  34. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  35. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 140
  36. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  37. 2023TableLlama: Towards Open Large Generalist Models for Tablescited by 136
  38. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133

50 most cited of 721 All 721 (JSON)