mlsanityData to 7 Oct 2026

LongRoPE2: Near-Lossless LLM Context Window Scaling

arXiv 2502.20082 ↗First posted 27 Feb 2025cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

LongRoPE2 is a novel approach that extends the effective context window of pre-trained large language models (LLMs) to the target length, while preserving the performance on the original shorter context window. This is achieved by three contributions: (1) a hypothesis that insufficient training in higher RoPE dimensions contributes to the persistent out-of-distribution (OOD) issues observed in existing methods; (2) an effective RoPE rescaling algorithm that adopts evolutionary search guided by "needle-driven" perplexity to address the insufficient training problem; (3) a mixed context window training approach that fine-tunes model weights to adopt rescaled RoPE for long-context sequences while preserving the short-context performance with the original RoPE. Extensive experiments on LLaMA3-8B and Phi3-mini-3.8B across various benchmarks validate the hypothesis and demonstrate the effectiveness of LongRoPE2. Remarkably, LongRoPE2 extends LLaMA3-8B to achieve a 128K effective context length while retaining over 98.5% of short-context performance, using only 10B tokens -- 80x fewer than Meta's approach, which fails to reach the target effective context length. Code will be available at https://github.com/microsoft/LongRoPE.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 41 of its 50 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

41 of 50 references.

References · 41

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  12. 2023StarCoder: may the source be with you!cited by 790
29 more
  1. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  2. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  3. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  4. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 437
  5. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  6. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  7. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  8. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  9. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  10. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  11. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  12. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  13. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 178
  14. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  15. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  16. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 140
  17. 2023Scaling Laws of RoPE-based Extrapolationcited by 124
  18. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 68
  19. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 61
  20. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 53
  21. 2024Base of RoPE Bounds Context Lengthcited by 49
  22. 2024Can Perplexity Reflect Large Language Model's Ability in Long Text Understanding?cited by 45
  23. 2024GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Modelscited by 42
  24. 2024LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelscited by 23
  25. 2023Robustifying State-space Models for Long Sequences via Approximate Diagonalizationcited by 18
  26. 2024BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Modelscited by 15
  27. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15
  28. 2024Multi-view Content-aware Indexing for Long Document Retrievalcited by 4
  29. 2024DCIS: Efficient Length Extrapolation of LLMs via Divide-and-Conquer Scaling Factor Searchcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 12.

21 papers.

Sort:

By area:

  1. 2025LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contextscited by 25
  2. 2025Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddingscited by 23
  3. 2025MemTool: Optimizing Short-Term Memory Management for Dynamic Tool Calling in LLM Agent Multi-Turn Conversationscited by 21
  4. 2025UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Modelscited by 13
  5. 2026Addressable Memory for Video World Modelscited by 11
  6. 2025Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaborationcited by 4
  7. 2025Q-RAG: Long Context Multi-step Retrieval via Value-based Embedder Trainingcited by 3
  8. 2026MrRoPE: Mixed-radix Rotary Position Embeddingcited by 3
  9. 2025Extending Audio Context for Long-Form Understanding in Large Audio-Language Modelscited by 2
  10. 2026Attention Needs to Focus: A Unified Perspective on Attention Allocationcited by 2
  11. 2025LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Trainingcited by 1
  12. 2025LLM2IR: simple unsupervised contrastive learning makes long-context LLM great retrievercited by 0
9 more
  1. 2026CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMscited by 0
  2. 2026From Buffers to Registers: Unlocking Fine-Grained FlashAttention with Hybrid-Bonded 3D NPU Co-Designcited by 0
  3. 2026AttentionRetriever: Attention Layers are Secretly Long Document Retrieverscited by 0
  4. 2026Short Data, Long Context: Distilling Positional Knowledge in Transformerscited by 0
  5. 2026Mitigating Position Bias in Transformers via Layer-Specific Positional Embedding Scalingcited by 0
  6. 2026Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scalingcited by 0
  7. 2026Learning how to Forget: Fine-tuning for Long-Context Sparse Attentioncited by 0
  8. 2026SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Modelscited by 0
  9. 2026StateTree: Enhancing Long-Term Dialogue Reasoning via Reinforcement Learningcited by 0