mlsanityData to 7 Oct 2026

Base of RoPE Bounds Context Length

arXiv 2405.14591 ↗First posted 23 May 2024cs.CLcited by 49

Not on the map. Why?

Abstract (arXiv)

Position embedding is a core component of current Large Language Models (LLMs). Rotary position embedding (RoPE), a technique that encodes the position information with a rotation matrix, has been the de facto choice for position embedding in many LLMs, such as the Llama series. RoPE has been further utilized to extend long context capability, which is roughly based on adjusting the textitbase parameter of RoPE to mitigate out-of-distribution (OOD) problems in position embedding. However, in this paper, we find that LLMs may obtain a superficial long-context ability based on the OOD theory. We revisit the role of RoPE in LLMs and propose a novel property of long-term decay, we derive that the textitbase of RoPE bounds context length: there is an absolute lower bound for the base value to obtain certain context length capability. Our work reveals the relationship between context length and RoPE base both theoretically and empirically, which may shed light on future long context training.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 34 of its 43 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

34 of 43 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  12. 2018Self-Attention with Relative Position Representationscited by 1,112
22 more
  1. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  2. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  3. 2024Yi: Open Foundation Models by 01.AIcited by 706
  4. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  5. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  6. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  7. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  8. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  9. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  10. 2024Rotary Position Embedding for Vision Transformercited by 226
  11. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  12. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  13. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  14. 2023Scaling Laws of RoPE-based Extrapolationcited by 124
  15. 2022A Length-Extrapolatable Transformercited by 123
  16. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  17. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  18. 2023Impact of Code Language Models on Automated Program Repaircited by 85
  19. 2024Can Perplexity Reflect Large Language Model's Ability in Long Text Understanding?cited by 45
  20. 2022Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysiscited by 39
  21. 2021SHAPE: Shifted Absolute Position Embedding for Transformerscited by 37
  22. 2019An Augmented Transformer Architecture for Natural Language Generation Taskscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 24; 2026: 13.

49 papers.

Sort:

By area:

  1. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  2. 2025Test-Time Training Done Rightcited by 121
  3. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  4. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  5. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
  6. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  7. 2025Baichuan-M1: Pushing the Medical Capability of Large Language Modelscited by 41
  8. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 38
  9. 2024ALR2: A Retrieve-then-Reason Framework for Long-context Question Answeringcited by 25
  10. 2025LongRoPE2: Near-Lossless LLM Context Window Scalingcited by 21
  11. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15
  12. 2024Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspectivecited by 13
37 more
  1. 2024On the token distance modeling ability of higher RoPE attention dimensioncited by 13
  2. 2025Mamba Modulation: On the Length Generalization of Mambacited by 13
  3. 2025Technical Report of TeleChat2, TeleChat2.5 and T1cited by 10
  4. 2024Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Modelscited by 9
  5. 2024Biology-Instructions: A Dataset and Benchmark for Multi-Omics Sequence Understanding Capability of Large Language Modelscited by 9
  6. 2024KV Shifting Attention Enhances Language Modelingcited by 7
  7. 2025SkyLadder: Better and Faster Pretraining via Context Window Schedulingcited by 6
  8. 2025Decoupling Positional and Symbolic Attention Behavior in Transformerscited by 6
  9. 2026Rotary Positional Embeddings as Phase Modulation: Theoretical Bounds on the RoPE Base for Long-Context Transformerscited by 6
  10. 2026RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provablycited by 6
  11. 2025Beyond In-Distribution Success: Scaling Curves of CoT Granularity for Language Model Generalizationcited by 5
  12. 2025Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generationcited by 4
  13. 2025HoPE: Hybrid of Position Embedding for Long Context Vision-Language Modelscited by 4
  14. 2025Rotary Masked Autoencoders are Versatile Learnerscited by 4
  15. 2025EgoMem: Lifelong Memory Agent for Full-duplex Omnimodal Modelscited by 4
  16. 2025LEDiT: Your Length-Extrapolatable Diffusion Transformer without Positional Encodingcited by 3
  17. 2026Mi:dm 2.0 Korea-centric Bilingual Language Modelscited by 3
  18. 2026MrRoPE: Mixed-radix Rotary Position Embeddingcited by 3
  19. 2026Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPEcited by 3
  20. 2025Context-aware Biases for Length Extrapolationcited by 2
  21. 2025Scaling Context, Not Parameters: Training a Compact 7B Language Model for Efficient Long-Context Processingcited by 2
  22. 2025ID-Align: RoPE-Conscious Position Remapping for Dynamic High-Resolution Adaptation in Vision-Language Modelscited by 2
  23. 2025SeqPE: Transformer with Sequential Position Encodingcited by 2
  24. 2025Predictable Compression Failures: Order Sensitivity and Information Budgeting for Evidence-Grounded Binary Adjudicationcited by 2
  25. 2025Review of Hallucination Understanding in Large Language and Vision Modelscited by 2
  26. 2025Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMscited by 2
  27. 2024What Kinds of Tokens Benefit from Distant Text? An Analysis on Long Context Language Modelingcited by 1
  28. 2024Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Modelscited by 1
  29. 2025A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)cited by 1
  30. 2026Remember to Forget: Gated Adaptive Positional Encodingcited by 1
  31. 2026LeRoPE: Learnable RoPE Frequencies Improve Language Modelingcited by 1
  32. 2026BSAT: B-Spline Adaptive Tokenizer for Long-Term Time Series Forecastingcited by 0
  33. 2026CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMscited by 0
  34. 2026Tango: Taming Visual Signals for Efficient Video Large Language Modelscited by 0
  35. 2026Transcript-Managed Transformers: Monotone Multi-Agent Collapse and Universality with Two Pop-Enabled Transcriptscited by 0
  36. 2026Disentangling the Expressivity of RoPEcited by 0
  37. 2026RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodingscited by 0