mlsanityData to 7 Oct 2026

LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis

arXiv 2607.06160 ↗First posted 7 Jul 2026cs.CL cs.AIcited by 0

On the map: Synthetic data and self-improvement for LLMs → Code and instruction data synthesis

Abstract (arXiv)

Synthesizing long-context supervised fine-tuning (SFT) data is a scalable way to enhance the long-context understanding of large language models (LLMs), yet existing approaches share three limitations: narrow task coverage, insufficient instruction difficulty, and a lack of faithfulness supervision. We propose LongCrafter, a structured synthesis framework that couples a hierarchical task taxonomy with an evidence-grounded pipeline. The taxonomy organizes long-context understanding into local/shallow and global/deep levels and yields 32 fine-grained task types that serve as a global generative prior. Guided by this taxonomy, LongCrafter constructs task-aligned long contexts, decomposes them into explicit evidence graphs that model cross-paragraph dependencies, and generates instruction--response pairs strictly grounded in the located evidence spans, ensuring both controllable difficulty and faithful, traceable reasoning. Models fine-tuned on LongCrafter data outperform all SFT baselines and even the official post-trained models on LongBench, LongBench~v2, and LooGLE across both Qwen2.5-7B and LLaMA-3.1-8B, with the largest gains on high-difficulty tasks. Further analysis shows that LongCrafter data is more diverse and better spread across difficulty levels, and that the trained models locate evidence robustly regardless of position, effectively mitigating the ``lost in the middle'' problem.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Code and instruction data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 19 of its 35 references that are in the map. First posted on arXiv 7 Jul 2026.

Builds on

19 of 35 references.

References · 19

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  6. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  7. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  8. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  9. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  10. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  11. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 140
  12. 2023Data Diversity Matters for Robust Instruction Tuningcited by 76
7 more
  1. 2020Diversity, Density, and Homogeneity: Quantitative Characteristic Metrics for Text Collectionscited by 21
  2. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  3. 2025GRADE: Generating multi-hop QA and fine-gRAined Difficulty matrix for RAG Evaluationcited by 7
  4. 2025LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructionscited by 6
  5. 2026GraphWalker: Agentic Knowledge Graph Question Answering via Synthetic Trajectory Curriculumcited by 6
  6. 2026Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shapingcited by 1
  7. 2026Scaling Multi-Hop Training Data via Graph-Constrained Path Selectioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning
  2. 2025Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation
  3. 2026A Decomposition Perspective to Long-context Reasoning for LLMs
  4. 2024MAmmoTH2: Scaling Instructions from the Web
  5. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Models
  6. 2025Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data
  7. 2025Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning
  8. 2026π2: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models