mlsanityData to 7 Oct 2026

LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data

arXiv 2502.12583 ↗First posted 18 Feb 2025cs.CLcited by 16

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

Despite the growing development of long-context large language models (LLMs), data-centric approaches relying on synthetic data have been hindered by issues related to faithfulness, which limit their effectiveness in enhancing model performance on tasks such as long-context reasoning and question answering (QA). These challenges are often exacerbated by misinformation caused by lack of verification, reasoning without attribution, and potential knowledge conflicts. We propose LongFaith, a novel pipeline for synthesizing faithful long-context reasoning instruction datasets. By integrating ground truth and citation-based reasoning prompts, we eliminate distractions and improve the accuracy of reasoning chains, thus mitigating the need for costly verification processes. We open-source two synthesized datasets, LongFaith-SFT and LongFaith-PO, which systematically address multiple dimensions of faithfulness, including verified reasoning, attribution, and contextual grounding. Extensive experiments on multi-hop reasoning datasets and LongBench demonstrate that models fine-tuned on these datasets significantly improve performance. Our ablation studies highlight the scalability and adaptability of the LongFaith pipeline, showcasing its broad applicability in developing long-context LLMs.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 40 of its 42 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

40 of 42 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks19 Dec
  2. 2024Large Language Models Can Self-Improve in Long-context Reasoning12 Nov

Other references · 38

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  11. 2024A Survey on LLM-as-a-Judgecited by 1,384
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
26 more
  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  2. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  3. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  4. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  5. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  6. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  7. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  8. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  9. 2018QuAC : Question Answering in Contextcited by 379
  10. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  11. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  12. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  13. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  14. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  15. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  16. 2024Make Your LLM Fully Utilize the Contextcited by 118
  17. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  18. 2024LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QAcited by 78
  19. 2023A Survey of Large Language Models Attributioncited by 71
  20. 2024Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generationcited by 63
  21. 2023End-to-End Beam Retrieval for Multi-Hop Question Answeringcited by 35
  22. 2024LongReward: Improving Long-context Large Language Models with AI Feedbackcited by 34
  23. 2024Learning to Plan and Generate Text with Citationscited by 28
  24. 2024Making Long-Context Language Models Better Multi-Hop Reasonerscited by 25
  25. 2024CoTAR: Chain-of-Thought Attribution Reasoning with Multi-level Granularitycited by 12
  26. 2024Learning Fine-Grained Grounded Citations for Attributed Large Language Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 8.

16 papers.

Sort:

By area:

  1. 2025SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimizationcited by 10
  2. 2025Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillationcited by 8
  3. 2025Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learningcited by 7
  4. 2025Synthesize-on-Graph: Knowledgeable Synthetic Data Generation for Continue Pre-training of Large Language Modelscited by 3
  5. 2025Chunks as Arms: Multi-Armed Bandit-Guided Sampling for Long-Context LLM Preference Optimizationcited by 3
  6. 2025GraphSearch: An Agentic Deep Searching Workflow for Graph Retrieval-Augmented Generationcited by 3
  7. 2025From Evidence to Trajectory: Abductive Reasoning Path Synthesis for Retrieval-Augmented Generation Agents Developmentcited by 2
  8. 2026Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shapingcited by 1
  9. 2026Probing How Scalable Table Data Enhances General Long-Context Reasoningcited by 1
  10. 2026A Decomposition Perspective to Long-context Reasoning for LLMscited by 1
  11. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 1
  12. 2025C2-Cite: Contextual-Aware Citation Generation for Attributed Large Language Modelscited by 0
4 more
  1. 2026Internalized Reasoning for Long-Context Visual Document Understandingcited by 0
  2. 2026π2: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Modelscited by 0
  3. 2026IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinkingcited by 0
  4. 2026Learning to Learn from Context: Synthetic Training from Perturbed Public Documentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion
  2. 2025LongReasonArena: A Long Reasoning Benchmark for Large Language Models
  3. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
  4. 2025Facilitating Long Context Understanding via Supervised Chain-of-Thought Reasoning
  5. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries
  6. 2025Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data
  7. 2024LongIns: A Challenging Long-context Instruction-based Exam for LLMs
  8. 2022Large Language Models are few(1)-shot Table Reasoners