mlsanityData to 14 Oct 2026

Teaching Language Models to Hallucinate Less with Synthetic Tasks

arXiv 2310.06827 ↗First posted 10 Oct 2023cs.CL cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) frequently hallucinate on abstractive summarization tasks such as document-based question-answering, meeting summarization, and clinical report generation, even though all necessary information is included in context. However, optimizing LLMs to hallucinate less on these tasks is challenging, as hallucination is hard to efficiently evaluate at each optimization step. In this work, we show that reducing hallucination on a synthetic task can also reduce hallucination on real-world downstream tasks. Our method, SynTra, first designs a synthetic task where hallucinations are easy to elicit and measure. It next optimizes the LLM's system message via prefix-tuning on the synthetic task, and finally transfers the system message to realistic, hard-to-optimize tasks. Across three realistic abstractive summarization tasks, SynTra reduces hallucination for two 13B-parameter LLMs using only a synthetic retrieval task for supervision. We also find that optimizing the system message rather than the model weights can be critical; fine-tuning the entire model on the synthetic task can counterintuitively increase hallucination. Overall, SynTra demonstrates that the extra flexibility of working with synthetic data can help mitigate undesired behaviors in practice.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 54 of its 66 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

54 of 66 references.

References · 54

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  10. 2017Deep reinforcement learning from human preferencescited by 3,912
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2020Learning to summarize from human feedbackcited by 2,588
38 more
  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  4. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  6. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  7. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  8. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  9. 2021Ethical and social risks of harm from Language Modelscited by 966
  10. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  11. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  13. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  14. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
  15. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  16. 2023Textbooks Are All You Needcited by 569
  17. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  18. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  19. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  20. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 336
  21. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  22. 2023How Language Model Hallucinations Can Snowballcited by 311
  23. 2023Evaluating Verifiability in Generative Search Enginescited by 303
  24. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  25. 2023Simple synthetic data reduces sycophancy in large language modelscited by 187
  26. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  27. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?cited by 154
  28. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 141
  29. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  30. 2021Entity-level Factual Consistency of Abstractive Text Summarizationcited by 118
  31. 2023ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generationcited by 118
  32. 2023Context-faithful Prompting for Large Language Modelscited by 80
  33. 2021The Factual Inconsistency Problem in Abstractive Text Summarization: A Surveycited by 79
  34. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 75
  35. 2023Symbol tuning improves in-context learning in language modelscited by 75
  36. 2023Automatic Evaluation of Attribution by Large Language Modelscited by 72
  37. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 72
  38. 2019Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generationcited by 62

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 17; 2025: 12; 2026: 1.

31 papers.

Sort:

By area:

  1. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 269
  2. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 159
  3. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
  4. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 102
  5. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Promptscited by 47
  6. 2024Multimodal Reasoning with Multimodal Knowledge Graphcited by 44
  7. 2024Evolutionary Large Language Model for Automated Feature Transformationcited by 29
  8. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  9. 2024Mitigating Large Language Model Hallucination with Faithful Finetuningcited by 24
  10. 2025BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretrainingcited by 21
  11. 2024Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)cited by 20
  12. 2024Improving Factuality in Large Language Models via Decoding-Time Hallucinatory and Truthful Comparatorscited by 19
19 more
  1. 2025Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learningcited by 11
  2. 2024FACTOID: FACtual enTailment fOr hallucInation Detectioncited by 10
  3. 2024Knowledge Verification to Nip Hallucination in the Budcited by 8
  4. 2024Generative AI Agents in Autonomous Machines: A Safety Perspectivecited by 7
  5. 2024Fictitious Synthetic Data Can Improve LLM Factuality via Prerequisite Learningcited by 7
  6. 2025Aligning Instruction Tuning with Pre-trainingcited by 7
  7. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 7
  8. 2024Information-Theoretic Distillation for Reference-less Summarizationcited by 6
  9. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  10. 2024Automatically Generating Numerous Context-Driven SFT Data for LLMs across Diverse Granularitycited by 4
  11. 2025The Curious Case of Factuality Finetuning: Models' Internal Beliefs Can Improve Factualitycited by 4
  12. 2025Virus Infection Attack on LLMs: Your Poisoning Can Spread "VIA" Synthetic Datacited by 4
  13. 2024Evaluating the Impact of Advanced LLM Techniques on AI-Lecture Tutors for a Robotics Coursecited by 3
  14. 2025A Scoping Review of Synthetic Data Generation by Language Models in Biomedical Research and Application: Data Utility and Quality Perspectivescited by 3
  15. 2025Finetune-RAG: Fine-Tuning Language Models to Resist Hallucination in Retrieval-Augmented Generationcited by 2
  16. 2025Data Swarms: Optimizable Generation of Synthetic Evaluation Datacited by 2
  17. 2026Multi-Head Attention Is a Multi-Player Gamecited by 2
  18. 2025HFuzzer: Testing Large Language Models for Package Hallucinations via Phrase-based Fuzzingcited by 1
  19. 2025TrueBrief: Faithful Summarization through Small Language Modelscited by 0