# Finetune-RAG: Fine-Tuning Language Models to Resist Hallucination in Retrieval-Augmented Generation

arXiv: https://arxiv.org/abs/2505.10792 · HTML: https://mlsanity.com/paper/2505.10792/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation.
- Builds on 28 of its 32 references that are in the map.
- First posted on arXiv 16 May 2025.

## Placed in

- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Builds on (most cited first, 28 of 28)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering](https://mlsanity.com/paper/2104.06378/) (2021)
- [Making Retrieval-Augmented Language Models Robust to Irrelevant Context](https://mlsanity.com/paper/2310.01558/) (2023)
- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models](https://mlsanity.com/paper/2401.01301/) (2024)
- [Seven Failure Points When Engineering a Retrieval Augmented Generation System](https://mlsanity.com/paper/2401.05856/) (2024)
- [The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediations](https://mlsanity.com/paper/2310.04988/) (2023)
- [R-Tuning: Instructing Large Language Models to Say `I Don't Know'](https://mlsanity.com/paper/2311.09677/) (2023)
- [Blended RAG: Improving RAG (Retriever-Augmented Generation) Accuracy with Semantic Search and Hybrid Query-Based Retrievers](https://mlsanity.com/paper/2404.07220/) (2024)
- [Trustworthiness in Retrieval-Augmented Generation Systems: A Survey](https://mlsanity.com/paper/2409.10102/) (2024)
- [On Learning to Summarize with Large Language Models as References](https://mlsanity.com/paper/2305.14239/) (2023)
- [StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurization](https://mlsanity.com/paper/2410.08815/) (2024)
- [Do LLMs Know about Hallucination? An Empirical Investigation of LLM's Hidden States](https://mlsanity.com/paper/2402.09733/) (2024)
- [Don't Forget to Connect! Improving RAG with Graph-based Reranking](https://mlsanity.com/paper/2405.18414/) (2024)
- [Deficiency of Large Language Models in Finance: An Empirical Examination of Hallucination](https://mlsanity.com/paper/2311.15548/) (2023)
- [Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Study](https://mlsanity.com/paper/2502.02481/) (2025)
- [Teaching Language Models to Hallucinate Less with Synthetic Tasks](https://mlsanity.com/paper/2310.06827/) (2023)
- [MedHalu: Hallucinations in Responses to Healthcare Queries by Large Language Models](https://mlsanity.com/paper/2409.19492/) (2024)
- [SRAG: Structured Retrieval-Augmented Generation for Multi-Entity Question Answering over Wikipedia Graph](https://mlsanity.com/paper/2503.01346/) (2025)
- [A Step Closer to Comprehensive Answers: Constrained Multi-Stage Question Decomposition with Large Language Models](https://mlsanity.com/paper/2311.07491/) (2023)
- [OpenRAG: Optimizing RAG End-to-End via In-Context Retrieval Learning](https://mlsanity.com/paper/2503.08398/) (2025)

## Built on by (most cited first, 2 of 2)

- [RAGAL: A Frugal, Fully Local Retrieval-Augmented Assistant for Technical Support at a Government Agency](https://mlsanity.com/paper/2607.18756/) (2026)
- [Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning](https://mlsanity.com/paper/2610.01936/) (2026)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) has emerged as a powerful framework to improve factuality in large language models (LLMs) by grounding their outputs in retrieved documents. However, ensuring perfect retrieval of relevant information remains challenging, and when irrelevant content is passed downstream to an LLM, it can lead to hallucinations. In this work, we propose Finetune-RAG, a simple and effective fine-tuning approach that features the first-of-its-kind RAG training dataset constructed to mimic real-world imperfections. Experimental results show that Finetune-RAG improves factual accuracy by 21.2% over the base model. We also propose Bench-RAG, an LLM-as-a-judge evaluation pipeline that stress tests models under realistic imperfect retrieval scenarios. Our codebase and dataset are fully open sourced for community use.