mlsanityData to 14 Oct 2026

Stick to What You Know: A Study of Knowledge-Aligned Supervised Fine-Tuning

arXiv 2608.30987 ↗First posted 31 Aug 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Supervised fine-tuning (SFT) trains a base language model to imitate target responses, and these targets may require knowledge the base model has not robustly internalized. We study this as a source of hallucinations and frame a group of mitigation methods as knowledge-aligned SFT: constraining SFT training targets to the base model's parametric knowledge. Under a unified setup, we compare existing generation-based and estimation-based knowledge-alignment methods and introduce two new variants: Evidence Rewrite, which verifies base-model generations using external evidence, and Recall Rewrite, which retains claims only when they can be consistently recalled by the base model. Experiments with Qwen 3 4B and OLMo 3 7B show that knowledge-aligned SFT can reduce factual hallucinations on WildHalu and Biography while largely preserving general capabilities. Recall Rewrite yields the strongest factuality gains and improves refusal behavior on UnknownBench. It thereby confirms that SFT targets beyond the base model's knowledge drive hallucination behavior.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 48 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

26 of 48 references.

References · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  5. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  6. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  7. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  8. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  9. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  10. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  11. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  12. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 264
14 more
  1. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
  2. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 178
  3. 2024Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantificationcited by 162
  4. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 132
  5. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 117
  6. 2024OLMES: A Standard for Language Model Evaluationscited by 106
  7. 2023Examining LLMs' Uncertainty Expression Towards Questions Outside Parametric Knowledgecited by 32
  8. 2024WildHallucinations: Evaluating Long-form Factuality in LLMs with Real-World Entity Queriescited by 26
  9. 2023KILM: Knowledge Injection into Encoder-Decoder Language Modelscited by 19
  10. 2026Empty Shelves or Lost Keys? Recall Is the Bottleneck for Parametric Factualitycited by 12
  11. 2024Fictitious Synthetic Data Can Improve LLM Factuality via Prerequisite Learningcited by 7
  12. 2026Why Fine-Tuning Encourages Hallucinations and How to Fix Itcited by 7
  13. 2026Self-Improving Pretraining: using post-trained models to pretrain better modelscited by 6
  14. 2025Listen to the Context: Towards Faithful Large Language Models for Retrieval Augmented Generation on Climate Questionscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Learning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigation
  2. 2025The Curious Case of Factuality Finetuning: Models' Internal Beliefs Can Improve Factuality
  3. 2024Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation
  4. 2026PRISM: Probing Reasoning, Instruction, and Source Memory in LLM Hallucinations
  5. 2026KARL: Mitigating Hallucinations in LLMs via Knowledge-Boundary-Aware Reinforcement Learning
  6. 2026Mitigating Prompt-Induced Hallucinations in Large Language Models via Structured Reasoning
  7. 2026SAKED: Mitigating Hallucination in Large Vision-Language Models via Stability-Aware Knowledge Enhanced Decoding
  8. 2025AutoRAG-LoRA: Hallucination-Triggered Knowledge Retuning via Lightweight Adapters