# Dehallucinating Parallel Context Extension for Retrieval-Augmented Generation

arXiv: https://arxiv.org/abs/2412.14905 · HTML: https://mlsanity.com/paper/2412.14905/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Hallucination detection in RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 27 of its 41 references that are in the map.
- First posted on arXiv 19 Dec 2024.

## Placed in

- Retrieval-augmented generation / [Hallucination detection in RAG](https://mlsanity.com/rag/hallucination-detection-in-rag/)

## Builds on (most cited first, 27 of 27)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [Active Retrieval Augmented Generation](https://mlsanity.com/paper/2305.06983/) (2023)
- [A Survey of Hallucination in Large Foundation Models](https://mlsanity.com/paper/2309.05922/) (2023)
- [Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy](https://mlsanity.com/paper/2305.15294/) (2023)
- [Chain-of-Verification Reduces Hallucination in Large Language Models](https://mlsanity.com/paper/2309.11495/) (2023)
- [Improving Text Embeddings with Large Language Models](https://mlsanity.com/paper/2401.00368/) (2023)
- [RAFT: Adapting Language Model to Domain Specific RAG](https://mlsanity.com/paper/2403.10131/) (2024)
- [Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration](https://mlsanity.com/paper/2402.00367/) (2024)
- [DocPrompting: Generating Code by Retrieving the Docs](https://mlsanity.com/paper/2207.05987/) (2022)
- [Make Your LLM Fully Utilize the Context](https://mlsanity.com/paper/2404.16811/) (2024)
- [Long-Context Language Modeling with Parallel Context Encoding](https://mlsanity.com/paper/2402.16617/) (2024)
- [Structured Prompting: Scaling In-Context Learning to 1,000 Examples](https://mlsanity.com/paper/2212.06713/) (2022)
- [Federated Recommendation via Hybrid Retrieval Augmented Generation](https://mlsanity.com/paper/2403.04256/) (2024)
- [Compressing Long Context for Enhancing RAG with AMR-based Concept Distillation](https://mlsanity.com/paper/2405.03085/) (2024)
- [Compositional API Recommendation for Library-Oriented Code Generation](https://mlsanity.com/paper/2402.19431/) (2024)
- [On Calibrated Model Uncertainty in Deep Learning](https://mlsanity.com/paper/2206.07795/) (2022)
- [Revisiting Parallel Context Windows: A Frustratingly Simple Alternative and Chain-of-Thought Deterioration](https://mlsanity.com/paper/2305.15262/) (2023)
- [Naive Bayes-based Context Extension for Large Language Models](https://mlsanity.com/paper/2403.17552/) (2024)
- [ParaICL: Towards Parallel In-Context Learning](https://mlsanity.com/paper/2404.00570/) (2024)

## Abstract (arXiv)

Large language models (LLMs) are susceptible to generating hallucinated information, despite the integration of retrieval-augmented generation (RAG). Parallel context extension (PCE) is a line of research attempting to effectively integrating parallel (unordered) contexts, while it still suffers from hallucinations when adapted to RAG scenarios. In this paper, we propose DePaC (Dehallucinating Parallel Context Extension), which alleviates the hallucination problem with context-aware negative training and information-calibrated aggregation. DePaC is designed to alleviate two types of in-context hallucination: fact fabrication (i.e., LLMs present claims that are not supported by the contexts) and fact omission (i.e., LLMs fail to present claims that can be supported by the contexts). Specifically, (1) for fact fabrication, we apply the context-aware negative training that fine-tunes the LLMs with negative supervisions, thus explicitly guiding the LLMs to refuse to answer when contexts are not related to questions; (2) for fact omission, we propose the information-calibrated aggregation which prioritizes context windows with higher information increment from their contexts. The experimental results on nine RAG tasks demonstrate that DePaC significantly alleviates the two types of hallucination and consistently achieves better performances on these tasks.