mlsanityData to 7 Oct 2026

Scaling Laws of RoPE-based Extrapolation

arXiv 2310.05209 ↗First posted 8 Oct 2023cs.CL cs.AIcited by 124

Not on the map. Why?

Abstract (arXiv)

The extrapolation capability of Large Language Models (LLMs) based on Rotary Position Embedding is currently a topic of considerable interest. The mainstream approach to addressing extrapolation with LLMs involves modifying RoPE by replacing 10000, the rotary base of θn=10000-2n/d in the original RoPE, with a larger value and providing longer fine-tuning text. In this work, we first observe that fine-tuning a RoPE-based LLM with either a smaller or larger base in pre-training context length could significantly enhance its extrapolation performance. After that, we propose textbftextitScaling Laws of RoPE-based Extrapolation, a unified framework from the periodic perspective, to describe the relationship between the extrapolation performance and base value as well as tuning context length. In this process, we also explain the origin of the RoPE-based extrapolation issue by textbftextitcritical dimension for extrapolation. Besides these observations and analyses, we achieve extrapolation up to 1 million context length within only 16K training length on LLaMA2 7B and 13B.

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). Builds on 39 of its 59 references that are in the map. First posted on arXiv 8 Oct 2023.

Builds on

39 of 59 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  12. 2020Longformer: The Long-Document Transformercited by 3,362
27 more
  1. 2023Qwen Technical Reportcited by 3,273
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2020Reformer: The Efficient Transformercited by 1,658
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  13. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  14. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  15. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  16. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  17. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  18. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  19. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  20. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  21. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  22. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 125
  23. 2022A Length-Extrapolatable Transformercited by 123
  24. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 110
  25. 2023Giraffe: Adventures in Expanding Context Lengths in LLMscited by 32
  26. 2023DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Trainingcited by 28
  27. 2016Towards Machine Comprehension of Spoken Content: Initial TOEFL Listening Comprehension Test by Machinecited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 54; 2025: 39; 2026: 26.

124 papers.

Sort:

By area:

  1. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  2. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  3. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  4. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  5. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  6. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  7. 2024Make Your LLM Fully Utilize the Contextcited by 118
  8. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  9. 2024FiT: Flexible Vision Transformer for Diffusion Modelcited by 84
  10. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communitiescited by 84
  11. 2023Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communicationcited by 78
  12. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
38 more
  1. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  2. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  3. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 61
  4. 2024ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilitiescited by 52
  5. 2024Base of RoPE Bounds Context Lengthcited by 49
  6. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  7. 2024Can Perplexity Reflect Large Language Model's Ability in Long Text Understanding?cited by 45
  8. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Timecited by 43
  9. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 39
  10. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  11. 2024Mind the Time: Temporally-Controlled Multi-Event Video Generationcited by 39
  12. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 38
  13. 2024JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Rerankingcited by 34
  14. 2023Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encodingcited by 32
  15. 2025Training Sparse Mixture Of Experts Text Embedding Modelscited by 32
  16. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 32
  17. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  18. 2023Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Usecited by 27
  19. 2024Resonance RoPE: Improving Context Length Generalization of Large Language Modelscited by 25
  20. 2024Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculumcited by 25
  21. 2024PyBench: Evaluating LLM Agent on various real-world coding taskscited by 24
  22. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMscited by 24
  23. 2024Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolationcited by 23
  24. 2024Apple Intelligence Foundation Language Modelscited by 22
  25. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  26. 2025LongRoPE2: Near-Lossless LLM Context Window Scalingcited by 21
  27. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 19
  28. 2026jina-embeddings-v5-text: Task-Targeted Embedding Distillationcited by 19
  29. 2024LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Modelscited by 18
  30. 2025FullPart: Generating each 3D Part at Full Resolutioncited by 17
  31. 2025SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modelingcited by 16
  32. 2024FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Modelcited by 15
  33. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15
  34. 2024UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generationcited by 14
  35. 2024CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Modelscited by 13
  36. 2024Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspectivecited by 13
  37. 2024ReAttention: Training-Free Infinite Context with Finite Attention Scopecited by 13
  38. 2024Towards LifeSpan Cognitive Systemscited by 13

50 most cited of 124 All 124 (JSON)