mlsanityData to 7 Oct 2026

Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data

arXiv 2510.25804 ↗First posted 29 Oct 2025cs.CLcited by 4

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Long-context language models unlock advanced capabilities in reasoning, code generation, and document summarization by leveraging dependencies across extended spans of text. However, a significant portion of readily available long-text data lacks meaningful long-distance dependencies; most spans can be predicted using only local context. Training on such data is inefficient, making careful data selection crucial. Therefore, we introduce LongFilter, a framework for curating training data tailored to long-context pretraining. LongFilter measures the information gain provided by extended context by contrasting model predictions under long-context versus short-context settings, thereby identifying samples where long-range dependencies are essential. Experiments with LLaMA-3-8B, extending its context length from 8K to 64K, show that LongFilter efficiently selects high-quality data and yields substantial improvements on benchmarks such as HELMET, LongBench, and RULER.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 31 of its 37 references that are in the map. First posted on arXiv 29 Oct 2025.

Builds on

31 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Qwen3 Technical Report14 May

Other references · 30

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  9. 2024Phi-4 Technical Reportcited by 675
  10. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  11. 2023Textbooks Are All You Needcited by 561
  12. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
18 more
  1. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  2. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  3. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  4. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  5. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  6. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  7. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  8. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  9. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  10. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  11. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  12. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  13. 2022Perplexed by Quality: A Perplexity-based Method for Adult and Harmful Content Detection in Multilingual Heterogeneous Web Datacited by 22
  14. 2024Extending LLMs' Context Window with 100 Samplescited by 20
  15. 2025ParallelComp: Parallel Long-Context Compressor for Length Extrapolationcited by 13
  16. 2022Automatic Document Selection for Efficient Encoder Pretrainingcited by 11
  17. 2024LongWanjuan: Towards Systematic Measurement for Long Text Qualitycited by 11
  18. 2025LongAttn: Selecting Long-context Training Data via Token-level Attentioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 1
  2. 2026The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corporacited by 0
  3. 2026What Should Data Teach? Moving Bottlenecks Across Circuit, Store, and Usecited by 0
  4. 2026What Shared Prefixes Hide: Trajectory Dropout for On-Policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench
  2. 2025Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning
  3. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
  4. 2025LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data
  5. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Models
  6. 2026LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis
  7. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation
  8. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack