# A Controlled Study on Long Context Extension and Generalization in LLMs

arXiv: https://arxiv.org/abs/2409.12181 · HTML: https://mlsanity.com/paper/2409.12181/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 39 references that are in the map.
- First posted on arXiv 18 Sep 2024.

## Builds on (most cited first, 22 of 22)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [Ring Attention with Blockwise Transformers for Near-Infinite Context](https://mlsanity.com/paper/2310.01889/) (2023)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models](https://mlsanity.com/paper/2309.12307/) (2023)
- [Landmark Attention: Random-Access Infinite Context Length for Transformers](https://mlsanity.com/paper/2305.16300/) (2023)
- [Compressing Context to Enhance Inference Efficiency of Large Language Models](https://mlsanity.com/paper/2310.06201/) (2023)
- [BookSum: A Collection of Datasets for Long-form Narrative Summarization](https://mlsanity.com/paper/2105.08209/) (2021)
- [In-Context Learning with Long-Context Models: An In-Depth Exploration](https://mlsanity.com/paper/2405.00200/) (2024)
- [Do Long-Range Language Models Actually Use Long-Range Context?](https://mlsanity.com/paper/2109.09115/) (2021)
- [A Benchmark for Learning to Translate a New Language from One Grammar Book](https://mlsanity.com/paper/2309.16575/) (2023)
- [CLEX: Continuous Length Extrapolation for Large Language Models](https://mlsanity.com/paper/2310.16450/) (2023)
- [In-Context Learning with Many Demonstration Examples](https://mlsanity.com/paper/2302.04931/) (2023)
- [LongHeads: Multi-Head Attention is Secretly a Long Context Processor](https://mlsanity.com/paper/2402.10685/) (2024)

## Built on by (most cited first, 26 of 26)

- [How to Train Long-Context Language Models (Effectively)](https://mlsanity.com/paper/2410.02660/) (2024)
- [Learning without training: The implicit dynamics of in-context learning](https://mlsanity.com/paper/2507.16003/) (2025)
- [Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets](https://mlsanity.com/paper/2604.02460/) (2026)
- [Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings](https://mlsanity.com/paper/2512.12167/) (2025)
- [Token Weighting for Long-Range Language Modeling](https://mlsanity.com/paper/2503.09202/) (2025)
- [From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models](https://mlsanity.com/paper/2504.06214/) (2025)
- [SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks](https://mlsanity.com/paper/2506.10424/) (2025)
- [The Role of Sparsity for Length Generalization in Transformers](https://mlsanity.com/paper/2502.16792/) (2025)
- [SkyLadder: Better and Faster Pretraining via Context Window Scheduling](https://mlsanity.com/paper/2503.15450/) (2025)
- [Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models](https://mlsanity.com/paper/2510.17196/) (2025)
- [100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?](https://mlsanity.com/paper/2505.19293/) (2025)
- [Revisiting Long-context Modeling from Context Denoising Perspective](https://mlsanity.com/paper/2510.05862/) (2025)
- [PERK: Long-Context Reasoning as Test-Time Learning](https://mlsanity.com/paper/2507.06415/) (2025)
- [Scaling Multi-Document Event Summarization: Evaluating Compression vs. Full-Text Approaches](https://mlsanity.com/paper/2502.06617/) (2025)
- [Learning When to Attend: Conditional Memory Access for Long-Context LLMs](https://mlsanity.com/paper/2603.17484/) (2026)
- [Simulating Human Memory with Language Models](https://mlsanity.com/paper/2605.25680/) (2026)
- [OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling](https://mlsanity.com/paper/2608.05141/) (2026)
- [FCPRAG: Fusion-Controller Parametric Retrieval-Augmented Generation for Stable Multi-Passage LoRA Injection](https://mlsanity.com/paper/2608.21750/) (2026)
- [TeamLLM: A Human-Like Team-Oriented Collaboration Framework for Multi-Step Contextualized Tasks](https://mlsanity.com/paper/2604.06765/) (2026)
- [Randomized YaRN Improves Length Generalization for Long-Context Reasoning](https://mlsanity.com/paper/2606.23687/) (2026)
- [How much do contextualized representations encode long-range context?](https://mlsanity.com/paper/2410.12292/) (2024)
- [Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation](https://mlsanity.com/paper/2504.18857/) (2025)
- [Gecko: An Efficient Neural Architecture Inherently Processing Sequences with Arbitrary Lengths](https://mlsanity.com/paper/2601.06463/) (2026)
- [AttentionRetriever: Attention Layers are Secretly Long Document Retrievers](https://mlsanity.com/paper/2602.12278/) (2026)
- [EndPrompt: Efficient Long-Context Extension via Terminal Anchoring](https://mlsanity.com/paper/2605.14589/) (2026)
- [Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension](https://mlsanity.com/paper/2608.10296/) (2026)

## Abstract (arXiv)

Broad textual understanding and in-context learning require language models that utilize full document contexts. Due to the implementation challenges associated with directly training long-context models, many methods have been proposed for extending models to handle long contexts. However, owing to differences in data and model classes, it has been challenging to compare these approaches, leading to uncertainty as to how to evaluate long-context performance and whether it differs from standard evaluation. We implement a controlled protocol for extension methods with a standardized evaluation, utilizing consistent base models and extension data. Our study yields several insights into long-context behavior. First, we reaffirm the critical role of perplexity as a general-purpose performance indicator even in longer-context tasks. Second, we find that current approximate attention methods systematically underperform across long-context tasks. Finally, we confirm that exact fine-tuning based methods are generally effective within the range of their extension, whereas extrapolation remains challenging. All codebases, models, and checkpoints will be made available open-source, promoting transparency and facilitating further research in this critical area of AI development.