# Merging Generated and Retrieved Knowledge for Open-Domain QA

arXiv: https://arxiv.org/abs/2310.14393 · HTML: https://mlsanity.com/paper/2310.14393/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 59 references that are in the map.
- First posted on arXiv 22 Oct 2023.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering](https://mlsanity.com/paper/2012.00955/) (2020)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [Generate rather than Retrieve: Large Language Models are Strong Context Generators](https://mlsanity.com/paper/2209.10063/) (2022)
- [Teaching Small Language Models to Reason](https://mlsanity.com/paper/2212.08410/) (2022)
- [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://mlsanity.com/paper/2110.07178/) (2021)
- [Distilling Knowledge from Reader to Retriever for Question Answering](https://mlsanity.com/paper/2012.04584/) (2020)
- [Generated Knowledge Prompting for Commonsense Reasoning](https://mlsanity.com/paper/2110.08387/) (2021)
- [Factuality Enhanced Language Models for Open-Ended Text Generation](https://mlsanity.com/paper/2206.04624/) (2022)
- [Large Language Models with Controllable Working Memory](https://mlsanity.com/paper/2211.05110/) (2022)

## Built on by (most cited first, 30 of 36)

- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [Knowledge Conflicts for LLMs: A Survey](https://mlsanity.com/paper/2403.08319/) (2024)
- [Evaluation of Retrieval-Augmented Generation: A Survey](https://mlsanity.com/paper/2405.07437/) (2024)
- [Trustworthiness in Retrieval-Augmented Generation Systems: A Survey](https://mlsanity.com/paper/2409.10102/) (2024)
- [Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models](https://mlsanity.com/paper/2410.07176/) (2024)
- [Enhancing Contextual Understanding in Large Language Models through Contrastive Decoding](https://mlsanity.com/paper/2405.02750/) (2024)
- [Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts When Knowledge Conflicts?](https://mlsanity.com/paper/2401.11911/) (2024)
- [AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledge](https://mlsanity.com/paper/2409.07394/) (2024)
- [A Survey on Knowledge-Oriented Retrieval-Augmented Generation](https://mlsanity.com/paper/2503.10677/) (2025)
- [PEDANTS: Cheap but Effective and Interpretable Answer Equivalence](https://mlsanity.com/paper/2402.11161/) (2024)
- [Towards Verifiable Text Generation with Evolving Memory and Self-Reflection](https://mlsanity.com/paper/2312.09075/) (2023)
- [To Generate or to Retrieve? On the Effectiveness of Artificial Contexts for Medical Open-Domain Question Answering](https://mlsanity.com/paper/2403.01924/) (2024)
- [Generative AI for Education (GAIED): Advances, Opportunities, and Challenges](https://mlsanity.com/paper/2402.01580/) (2024)
- [LLMs' Reading Comprehension Is Affected by Parametric Knowledge and Struggles with Hypothetical Statements](https://mlsanity.com/paper/2404.06283/) (2024)
- [APTNESS: Incorporating Appraisal Theory and Emotion Support Strategies for Empathetic Response Generation](https://mlsanity.com/paper/2407.21048/) (2024)
- [Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions](https://mlsanity.com/paper/2410.02028/) (2024)
- [LLM-Lasso: A Robust Framework for Domain-Informed Feature Selection and Regularization](https://mlsanity.com/paper/2502.10648/) (2025)
- [CoCoA: Confidence and Context-Aware Adaptive Decoding for Resolving Knowledge Conflicts in Large Language Models](https://mlsanity.com/paper/2508.17670/) (2025)
- [Evaluating the External and Parametric Knowledge Fusion of Large Language Models](https://mlsanity.com/paper/2405.19010/) (2024)
- [MOKA: Moral Knowledge Augmentation for Moral Event Extraction](https://mlsanity.com/paper/2311.09733/) (2023)
- [Helpful Agent Meets Deceptive Judge: Understanding Vulnerabilities in Agentic Workflows](https://mlsanity.com/paper/2506.03332/) (2025)
- [QRMeM: Unleash the Length Limitation through Question then Reflection Memory Mechanism](https://mlsanity.com/paper/2406.13167/) (2024)
- [Hallucinate at the Last in Long Response Generation: A Case Study on Long Document Summarization](https://mlsanity.com/paper/2505.15291/) (2025)
- [Familiarity-Aware Evidence Compression for Retrieval-Augmented Generation](https://mlsanity.com/paper/2409.12468/) (2024)
- [Exploring the Impact of Instruction-Tuning on LLM's Susceptibility to Misinformation](https://mlsanity.com/paper/2507.18203/) (2025)
- [HealthContradict: Evaluating Biomedical Knowledge Conflicts in Language Models](https://mlsanity.com/paper/2512.02299/) (2025)
- [Optimizing Knowledge Integration in Retrieval-Augmented Generation with Self-Selection](https://mlsanity.com/paper/2502.06148/) (2025)
- [Fine-Tuning Large Language Models and Evaluating Retrieval Methods for Improved Question Answering on Building Codes](https://mlsanity.com/paper/2505.04666/) (2025)
- [CC-VQA: Conflict- and Correlation-Aware Method for Mitigating Knowledge Conflict in Knowledge-Based Visual Question Answering](https://mlsanity.com/paper/2602.23952/) (2026)
- [Improving Generated and Retrieved Knowledge Combination Through Zero-shot Generation](https://mlsanity.com/paper/2412.18800/) (2024)

## Abstract (arXiv)

Open-domain question answering (QA) systems are often built with retrieval modules. However, retrieving passages from a given source is known to suffer from insufficient knowledge coverage. Alternatively, prompting large language models (LLMs) to generate contextual passages based on their parametric knowledge has been shown to improve QA performance. Yet, LLMs tend to "hallucinate" content that conflicts with the retrieved knowledge. Based on the intuition that answers supported by both sources are more likely to be correct, we propose COMBO, a Compatibility-Oriented knowledge Merging for Better Open-domain QA framework, to effectively leverage the two sources of information. Concretely, we match LLM-generated passages with retrieved counterparts into compatible pairs, based on discriminators trained with silver compatibility labels. Then a Fusion-in-Decoder-based reader model handles passage pairs to arrive at the final answer. Experiments show that COMBO outperforms competitive baselines on three out of four tested open-domain QA benchmarks. Further analysis reveals that our proposed framework demonstrates greater efficacy in scenarios with a higher degree of knowledge conflicts.