Balancing Truthfulness and Informativeness with Uncertainty-Aware Instruction Fine-Tuning

Instruction fine-tuning (IFT) can increase the informativeness of large language models (LLMs), but may reduce their truthfulness. This trade-off arises because IFT steers LLMs to generate responses containing long-tail knowledge that was not well covered during pre-training. As a result, models become more informative but less accurate when generalizing to unseen tasks. In this paper, we empirically demonstrate how unfamiliar knowledge in IFT datasets can negatively affect the truthfulness of LLMs, and we introduce two new IFT paradigms, UNITcut and UNITref, to address this issue. UNITcut identifies and removes unfamiliar knowledge from IFT datasets to mitigate its impact on model truthfulness, whereas UNITref trains LLMs to recognize their uncertainty and explicitly indicate it at the end of their responses. Our experiments show that UNITcut substantially improves LLM truthfulness, while UNITref maintains high informativeness and reduces hallucinations by distinguishing between confident and uncertain statements.