mlsanityData to 7 Oct 2026

LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens

arXiv 2402.13753 ↗First posted 21 Feb 2024cs.CLcited by 302

Not on the map. Why?

Cited by 32 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large context window is a desirable feature in large language models (LLMs). However, due to high fine-tuning costs, scarcity of long texts, and catastrophic values introduced by new token positions, current extended context windows are limited to around 128k tokens. This paper introduces LongRoPE that, for the first time, extends the context window of pre-trained LLMs to an impressive 2048k tokens, with up to only 1k fine-tuning steps at within 256k training lengths, while maintaining performance at the original short context window. This is achieved by three key innovations: (i) we identify and exploit two forms of non-uniformities in positional interpolation through an efficient search, providing a better initialization for fine-tuning and enabling an 8x extension in non-fine-tuning scenarios; (ii) we introduce a progressive extension strategy that first fine-tunes a 256k length LLM and then conducts a second positional interpolation on the fine-tuned extended LLM to achieve a 2048k context window; (iii) we readjust LongRoPE on 8k length to recover the short context window performance. Extensive experiments on LLaMA2 and Mistral across various tasks demonstrate the effectiveness of our method. Models extended via LongRoPE retain the original architecture with minor modifications to the positional embedding, and can reuse most pre-existing optimizations.

Key facts. Cited by 302 papers in the map (data through 7 Oct 2026). Builds on 22 of its 26 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

22 of 26 references.

References · 22

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
10 more
  1. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  2. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  3. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  4. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  5. 2023Augmenting Language Models with Long-Term Memorycited by 186
  6. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  7. 2023Scaling Laws of RoPE-based Extrapolationcited by 124
  8. 2024Long Context Compression with Activation Beaconcited by 101
  9. 2023Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruningcited by 26
  10. 2023SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstractioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 79; 2025: 123; 2026: 100.

302 papers.

Sort:

By area:

  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  3. 2024Long Context Transfer from Language to Visioncited by 502
  4. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  5. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  6. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  7. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  8. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  9. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  10. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  11. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  12. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
38 more
  1. 2024Make Your LLM Fully Utilize the Contextcited by 118
  2. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  3. 2024AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agentscited by 100
  4. 2025DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Processcited by 97
  5. 2024ImageFolder: Autoregressive Image Generation with Folded Tokenscited by 93
  6. 2025StreamingVLM: Real-Time Understanding for Infinite Video Streamscited by 89
  7. 2024From Isolated Conversations to Hierarchical Schemas: Dynamic Tree Memory Representation for LLMscited by 77
  8. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compressioncited by 71
  9. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  10. 2025RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformerscited by 63
  11. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  12. 2024Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesiscited by 55
  13. 2025Edit Flows: Flow Matching with Edit Operationscited by 53
  14. 2024On Large Language Model Continual Unlearningcited by 51
  15. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 51
  16. 2024LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answeringcited by 49
  17. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 47
  18. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansioncited by 47
  19. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  20. 2024GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Modelscited by 42
  21. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  22. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 38
  23. 2024Insights into LLM Long-Context Failures: When Transformers Know but Don't Tellcited by 36
  24. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
  25. 2025Spatia: Video Generation with Updatable Spatial Memorycited by 33
  26. 2024Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformerscited by 32
  27. 2025A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrievalcited by 31
  28. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  29. 2024Training Tensor Attention Efficiently: From Cubic to Almost Linear Timecited by 29
  30. 2024MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projectioncited by 29
  31. 2025Parallel Scaling Law for Language Modelscited by 29
  32. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  33. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 28
  34. 2024LongGenBench: Long-context Generation Benchmarkcited by 27
  35. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 26
  36. 2025CommVQ: Commutative Vector Quantization for KV Cache Compressioncited by 25
  37. 2025LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contextscited by 25
  38. 2025Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddingscited by 23

50 most cited of 302 All 302 (JSON)