mlsanityData to 7 Oct 2026

LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs

arXiv 2503.02502 ↗First posted 4 Mar 2025cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Long-context modeling has drawn more and more attention in the area of Large Language Models (LLMs). Continual training with long-context data becomes the de-facto method to equip LLMs with the ability to process long inputs. However, it still remains an open challenge to measure the quality of long-context training data. To address this issue, we propose a Long-context data selection framework with Attention-based Dependency Measurement (LADM), which can efficiently identify high-quality long-context data from a large-scale, multi-domain pre-training corpus. LADM leverages the retrieval capabilities of the attention mechanism to capture contextual dependencies, ensuring a comprehensive quality measurement of long-context data. Experimental results show that our LADM framework significantly boosts the performance of LLMs on multiple long-context tasks with only 1B tokens for continual training.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 35 of its 54 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

35 of 54 references.

References · 35

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  10. 2023Mistral 7Bcited by 3,065
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
23 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  4. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  5. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  6. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  7. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  8. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  9. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  10. 2021Efficient Attentions for Long Document Summarizationcited by 342
  11. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 323
  12. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  13. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  14. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  15. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  16. 2024Make Your LLM Fully Utilize the Contextcited by 118
  17. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 97
  18. 2024Can Perplexity Reflect Large Language Model's Ability in Long Text Understanding?cited by 45
  19. 2023Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Trainingcited by 31
  20. 2025LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimizationcited by 24
  21. 2024From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Datacited by 18
  22. 2023Structured Packing in LLM Training Improves Long Context Utilizationcited by 17
  23. 2023CoCA: Fusing Position Embedding with Collinear Constrained Attention in Transformers for Long Context Window Extendingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 5.

11 papers.

  1. 2025Look Again, Think Slowly: Enhancing Visual Reflection in Vision-Language Modelscited by 16
  2. 2025CROP: Contextual Region-Oriented Visual Token Pruningcited by 14
  3. 2025Structured Memory Mechanisms for Stable Context Representation in Large Language Modelscited by 9
  4. 2025ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learningcited by 9
  5. 2025TRIM: Token-wise Attention-Derived Saliency for Data-Efficient Instruction Tuningcited by 3
  6. 2026DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Modelscited by 2
  7. 2025Short-Context Dominance: How Much Local Context Natural Language Actually Needs?cited by 1
  8. 2026Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledgecited by 1
  9. 2026UT-ACA: Uncertainty-Triggered Adaptive Context Allocation for Long-Context Inferencecited by 0
  10. 2026The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corporacited by 0
  11. 2026What Should Data Teach? Moving Bottlenecks Across Circuit, Store, and Usecited by 0