mlsanityData to 14 Oct 2026

The Curious Case of Factuality Finetuning: Models' Internal Beliefs Can Improve Factuality

arXiv 2507.08371 ↗First posted 11 Jul 2025cs.CLcited by 4

On the map: Synthetic data and self-improvement for LLMs → Self-improving MLLMs and synthetic data

Abstract (arXiv)

Language models are prone to hallucination - generating text that is factually incorrect. Finetuning models on high-quality factual information can potentially reduce hallucination, but concerns remain; obtaining factual gold data can be expensive and training on correct but unfamiliar data may potentially lead to even more downstream hallucination. What data should practitioners finetune on to mitigate hallucinations in language models? In this work, we study the relationship between the factuality of finetuning data and the prevalence of hallucinations in long-form generation tasks. Counterintuitively, we find that finetuning on factual gold data is not as helpful as finetuning on model-generated data that models believe to be factual. Next, we evaluate filtering strategies applied on both factual gold data and model-generated data, and find that finetuning on model-generated data that is filtered by models' own internal judgments often leads to better overall factuality compared to other configurations: training on gold data filtered by models' judgments, training on gold data alone, or training on model-generated data that is supported by gold data. These factuality improvements transfer across three domains we study, suggesting that a models' own beliefs can provide a powerful signal for factuality.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Self-improving MLLMs and synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 43 of its 44 references that are in the map. First posted on arXiv 11 Jul 2025.

Builds on

43 of 44 references.

References · 43

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  7. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  8. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  9. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  10. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  11. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  12. 2022Quantifying Memorization Across Neural Language Modelscited by 763
31 more
  1. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
  2. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  3. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 257
  4. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
  5. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
  6. 2023Fine-tuning Language Models for Factualitycited by 221
  7. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 206
  8. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  9. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  10. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 153
  11. 2024Fine-grained Hallucination Detection and Editing for Language Modelscited by 149
  12. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 117
  13. 2022Locally Typical Samplingcited by 111
  14. 2023R-Tuning: Instructing Large Language Models to Say `I Don't Know'cited by 111
  15. 2021Decontextualization: Making Sentences Stand-Alonecited by 102
  16. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinatecited by 87
  17. 2023Alignment for Honestycited by 74
  18. 2023Do Androids Know They're Only Dreaming of Electric Sheep?cited by 67
  19. 2024Factuality of Large Language Models: A Surveycited by 66
  20. 2023Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?cited by 62
  21. 2023The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Modelscited by 55
  22. 2023Personas as a Way to Model Truthfulness in Language Modelscited by 47
  23. 2024FLAME: Factuality-Aware Alignment for Large Language Modelscited by 47
  24. 2024Molecular Facts: Desiderata for Decontextualization in LLM Fact Verificationcited by 44
  25. 2024Training Language Models to Generate Text with Citations via Fine-grained Rewardscited by 42
  26. 2023Language Models Hallucinate, but May Excel at Fact Verificationcited by 37
  27. 2024Calibrating Long-form Generations from Large Language Modelscited by 32
  28. 2024Understanding Finetuning for Factual Knowledge Extractioncited by 32
  29. 2023Teaching Language Models to Hallucinate Less with Synthetic Taskscited by 31
  30. 2024FactCheckmate: Preemptively Detecting and Mitigating Hallucinations in LMscited by 11
  31. 2024Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectabilitycited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 27
  2. 2026Illusions of Confidence? Diagnosing LLM Truthfulness via Neighborhood Consistencycited by 2
  3. 2026Purging the Gray Zone: Latent-Geometric Denoising for Precise Knowledge Boundary Awarenesscited by 0
  4. 2026Does Finetuning with Scientific Data Increase Hallucinations? A Multi-domain Factuality Evaluation of LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models
  2. 2025Are the Hidden States Hiding Something? Testing the Limits of Factuality-Encoding Capabilities in LLMs
  3. 2024On the Universal Truthfulness Hyperplane Inside LLMs
  4. 2025Theoretical Foundations and Mitigation of Hallucination in Large Language Models
  5. 2025OnionEval: An Unified Evaluation of Fact-conflicting Hallucination for Small-Large Language Models
  6. 2025MultiHal: Multilingual Dataset for Knowledge-Graph Grounded Evaluation of LLM Hallucinations
  7. 2026Unmasking Hallucinations: A Causal Graph-Attention Perspective on Factual Reliability in Large Language Models
  8. 2024Banishing LLM Hallucinations Requires Rethinking Generalization