mlsanityData to 7 Oct 2026

Resonance RoPE: Improving Context Length Generalization of Large Language Models

arXiv 2403.00071 ↗First posted 29 Feb 2024cs.CL cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

This paper addresses the challenge of train-short-test-long (TSTL) scenarios in Large Language Models (LLMs) equipped with Rotary Position Embedding (RoPE), where models pre-trained on shorter sequences face difficulty with out-of-distribution (OOD) token positions in longer sequences. We introduce Resonance RoPE, a novel approach designed to narrow the generalization gap in TSTL scenarios by refining the interpolation of RoPE features for OOD positions, significantly improving the model performance without additional online computational costs. Furthermore, we present PosGen, a new synthetic benchmark specifically designed for fine-grained behavior analysis in TSTL scenarios, aiming to isolate the constantly increasing difficulty of token generation on long contexts from the challenges of recognizing new token positions. Our experiments on synthetic tasks show that after applying Resonance RoPE, Transformers recognize OOD position better and more robustly. Our extensive LLM experiments also show superior performance after applying Resonance RoPE to the current state-of-the-art RoPE scaling method, YaRN, on both upstream language modeling tasks and a variety of downstream long-text applications.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 22 of its 32 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

22 of 32 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  10. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  11. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  12. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
10 more
  1. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  2. 2021Efficient Attentions for Long Document Summarizationcited by 342
  3. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  4. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  5. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  6. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  7. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  8. 2023Scaling Laws of RoPE-based Extrapolationcited by 124
  9. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  10. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 10; 2026: 6.

25 papers.

Sort:

By area:

  1. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
  2. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 47
  3. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargementcited by 36
  4. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  5. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  6. 2024LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelscited by 23
  7. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15
  8. 2024E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoningcited by 13
  9. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  10. 2025Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddingscited by 10
  11. 2024A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contextscited by 9
  12. 2025UltraFlux: Data-Model Co-Design for High-quality Native 4K Text-to-Image Generation across Diverse Aspect Ratioscited by 6
13 more
  1. 2025Resona: Improving Context Copying in Linear Recurrence Models with Retrievalcited by 5
  2. 2025Scale-invariant Attentioncited by 5
  3. 2025Long-Short Alignment for Effective Long-Context Modeling in LLMscited by 3
  4. 2025Improving GUI Grounding with Explicit Position-to-Coordinate Mappingcited by 3
  5. 2026MrRoPE: Mixed-radix Rotary Position Embeddingcited by 3
  6. 2026Frayed RoPE and Long Inputs: A Geometric Perspectivecited by 3
  7. 2026Exploration of Fast-Slow Latent Recurrence for Train-Short, Test-Long Generalizationcited by 2
  8. 2025Multi-Model Synthetic Training for Mission-Critical Small Language Modelscited by 1
  9. 2025Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Modelscited by 0
  10. 2025Of All StrIPEs: Investigating Structure-informed Positional Encoding for Efficient Music Generationcited by 0
  11. 2026Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptationcited by 0
  12. 2026Disentangling the Expressivity of RoPEcited by 0
  13. 2026RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodingscited by 0