# Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learning

arXiv: https://arxiv.org/abs/2505.16483 · HTML: https://mlsanity.com/paper/2505.16483/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 60 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 30 of 38)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [Trusting Your Evidence: Hallucinate Less with Context-aware Decoding](https://mlsanity.com/paper/2305.14739/) (2023)
- [Knowledge Conflicts for LLMs: A Survey](https://mlsanity.com/paper/2403.08319/) (2024)
- [MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents](https://mlsanity.com/paper/2404.10774/) (2024)
- [Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA](https://mlsanity.com/paper/2406.17419/) (2024)
- [DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications](https://mlsanity.com/paper/1711.05073/) (2017)
- [Sentence Simplification with Deep Reinforcement Learning](https://mlsanity.com/paper/1703.10931/) (2017)
- [FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"](https://mlsanity.com/paper/2410.03727/) (2024)
- [SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents](https://mlsanity.com/paper/2305.13040/) (2023)
- [Retrieval-Augmented Generation with Conflicting Evidence](https://mlsanity.com/paper/2504.13079/) (2025)
- [A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models](https://mlsanity.com/paper/2406.11289/) (2024)
- [Context-DPO: Aligning Language Models for Context-Faithfulness](https://mlsanity.com/paper/2412.15280/) (2024)
- [LongReward: Improving Long-context Large Language Models with AI Feedback](https://mlsanity.com/paper/2410.21252/) (2024)
- [Teaching Language Models to Hallucinate Less with Synthetic Tasks](https://mlsanity.com/paper/2310.06827/) (2023)
- [CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systems](https://mlsanity.com/paper/2404.02103/) (2024)
- [VCSUM: A Versatile Chinese Meeting Summarization Dataset](https://mlsanity.com/paper/2305.05280/) (2023)

## Built on by (most cited first, 11 of 11)

- [RhinoInsight: Improving Deep Research through Control Mechanisms for Model Behavior and Context](https://mlsanity.com/paper/2511.18743/) (2025)
- [Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models](https://mlsanity.com/paper/2602.05897/) (2026)
- [From Context to EDUs: Faithful and Structured Context Compression via Elementary Discourse Unit Decomposition](https://mlsanity.com/paper/2512.14244/) (2025)
- [Copy-Paste to Mitigate Large Language Model Hallucinations](https://mlsanity.com/paper/2510.00508/) (2025)
- [KARL: Mitigating Hallucinations in LLMs via Knowledge-Boundary-Aware Reinforcement Learning](https://mlsanity.com/paper/2604.22779/) (2026)
- [Structure-Token Evidence-Anchored Reasoning for Scientific Chart Understanding](https://mlsanity.com/paper/2609.13267/) (2026)
- [Capability-Routed Visual Retrieval and Evidence Threading for Long-Context Document Question Answering](https://mlsanity.com/paper/2609.13268/) (2026)
- [Learning to Learn from Context: Synthetic Training from Perturbed Public Documents](https://mlsanity.com/paper/2609.33642/) (2026)
- [TraceICL: Thread-Resolved Demonstration Curation with Consistency Calibration for Robust Multimodal In-Context Learning](https://mlsanity.com/paper/2610.12516/) (2026)
- [TerraAgent: Demonstration-Grounded Multi-Agent Reasoning for Long-Context Remote Sensing Visual Question Answering](https://mlsanity.com/paper/2610.12517/) (2026)
- [Reflective Prompt Optimization of a Deep-Research Agent: A Topic-Disjoint, Multi-Seed Protocol for Measuring Transferable Gains and Prompt-Level Overfitting](https://mlsanity.com/paper/2610.12601/) (2026)

## Abstract (arXiv)

Teaching large language models (LLMs) to be faithful in the provided context is crucial for building reliable information-seeking systems. Therefore, we propose a systematic framework, CANOE, to reduce faithfulness hallucinations of LLMs across different downstream tasks without human annotations. Specifically, we first synthesize short-form question-answering (QA) data with four diverse tasks to construct high-quality and easily verifiable training data without human annotation. Also, we propose Dual-GRPO, a rule-based reinforcement learning method that includes three tailored rule-based rewards derived from synthesized short-form QA data, while simultaneously optimizing both short-form and long-form response generation. Notably, Dual-GRPO eliminates the need to manually label preference data to train reward models and avoids over-optimizing short-form generation when relying only on the synthesized short-form QA data. Experimental results show that CANOE greatly improves the faithfulness of LLMs across 11 different tasks, even outperforming the most advanced LLMs, e.g., GPT-4o and OpenAI o1.