mlsanityData to 7 Oct 2026

YaRN: Efficient Context Window Extension of Large Language Models

arXiv 2309.00071 ↗First posted 31 Aug 2023cs.CL cs.AI cs.LGcited by 584

Not on the map. Why?

Cited by 64 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Rotary Position Embeddings (RoPE) have been shown to effectively encode positional information in transformer-based language models. However, these models fail to generalize past the sequence length they were trained on. We present YaRN (Yet another RoPE extensioN method), a compute-efficient method to extend the context window of such models, requiring 10x less tokens and 2.5x less training steps than previous methods. Using YaRN, we show that LLaMA models can effectively utilize and extrapolate to context lengths much longer than their original pre-training would allow, while also surpassing previous the state-of-the-art at context window extension. In addition, we demonstrate that YaRN exhibits the capability to extrapolate beyond the limited context of a fine-tuning dataset. Code is available at https://github.com/jquesnelle/yarn

Key facts. Cited by 584 papers in the map (data through 7 Oct 2026). Builds on 22 of its 44 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

22 of 44 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,105
10 more
  1. 2018Self-Attention with Relative Position Representationscited by 1,112
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  3. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  4. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  5. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  6. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  7. 2017Convolutional Sequence to Sequence Learningcited by 550
  8. 2021Efficient Attentions for Long Document Summarizationcited by 342
  9. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  10. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 144; 2025: 186; 2026: 230.

584 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2024Qwen2 Technical Reportcited by 2,076
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  7. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  8. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  10. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  11. 2024DeepSeek-V3 Technical Reportcited by 724
  12. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
38 more
  1. 2025Kimi K2: Open Agentic Intelligencecited by 523
  2. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  3. 2024Long Context Transfer from Language to Visioncited by 502
  4. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  5. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  6. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 401
  7. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  8. 2023Llemma: An Open Language Model For Mathematicscited by 385
  9. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  10. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 349
  11. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  12. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  13. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  14. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  15. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  16. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  17. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  18. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  19. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  20. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  21. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  22. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  23. 2023Efficient Large Language Models: A Surveycited by 183
  24. 2025World Simulation with Video Foundation Models for Physical AIcited by 183
  25. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 176
  26. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  27. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  28. 2025Qwen2.5-1M Technical Reportcited by 158
  29. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  30. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  31. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  32. 2025Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correctioncited by 140
  33. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  34. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 136
  35. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  36. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  37. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiTcited by 130
  38. 2023Scaling Laws of RoPE-based Extrapolationcited by 124

50 most cited of 584 All 584 (JSON)