mlsanityData to 14 Oct 2026

LongAttn: Selecting Long-context Training Data via Token-level Attention

arXiv 2502.16860 ↗First posted 24 Feb 2025cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

With the development of large language models (LLMs), there has been an increasing need for significant advancements in handling long contexts. To enhance long-context capabilities, constructing high-quality training data with long-range dependencies is crucial. Existing methods to select long-context data often rely on sentence-level analysis, which can be greatly optimized in both performance and efficiency. In this paper, we propose a novel token-level framework, LongAttn, which leverages the self-attention mechanism of LLMs to measure the long-range dependencies for the data. By calculating token-level dependency strength and distribution uniformity of token scores, LongAttn effectively quantifies long-range dependencies, enabling more accurate and efficient data selection. We filter LongABC-32K from open-source long-context datasets (ArXiv, Book, and Code). Through our comprehensive experiments, LongAttn has demonstrated its excellent effectiveness, scalability, and efficiency. To facilitate future research in long-context data, we released our code and the high-quality long-context training data LongABC-32K.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 47 of its 49 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

47 of 49 references.

References · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2023Qwen Technical Reportcited by 3,344
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
35 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  4. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  6. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  7. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  8. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  9. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  10. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  11. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  12. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  13. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  14. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  15. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  16. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 198
  17. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  18. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  19. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 152
  20. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  21. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 136
  22. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  23. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  24. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 106
  25. 2019On the Use of ArXiv as a Datasetcited by 103
  26. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  27. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 63
  28. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 62
  29. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 48
  30. 2024Long Context Alignment with Short Instructions and Synthesized Positionscited by 22
  31. 2024GATEAU: Selecting Influential Samples for Long Context Alignmentcited by 20
  32. 2024Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Modelscited by 16
  33. 2024What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practicescited by 14
  34. 2024LongWanjuan: Towards Systematic Measurement for Long Text Qualitycited by 11
  35. 2024Bootstrap Your Own Context Lengthcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2025Structured Memory Mechanisms for Stable Context Representation in Large Language Modelscited by 9
  2. 2025Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaborationcited by 7
  3. 2025Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Datacited by 4
  4. 2026From Interpretability to Performance: Optimizing Retrieval Heads for Long-Context Language Modelscited by 4
  5. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 2
  6. 2025Modular Techniques for Synthetic Long-Context Data Generation in Language Model Training and Evaluationcited by 1
  7. 2026The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corporacited by 1
  8. 2026Fast and Faithful: Real-Time Verification for Long-Document Retrieval-Augmented Generation Systemscited by 0
  9. 2026What Should Data Teach? Moving Bottlenecks Across Circuit, Store, and Usecited by 0