mlsanityData to 14 Oct 2026

Towards Better Generalization in Open-Domain Question Answering by Mitigating Context Memorization

arXiv 2404.01652 ↗First posted 2 Apr 2024cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Open-domain Question Answering (OpenQA) aims at answering factual questions with an external large-scale knowledge corpus. However, real-world knowledge is not static; it updates and evolves continually. Such a dynamic characteristic of knowledge poses a vital challenge for these models, as the trained models need to constantly adapt to the latest information to make sure that the answers remain accurate. In addition, it is still unclear how well an OpenQA model can transfer to completely new knowledge domains. In this paper, we investigate the generalization performance of a retrieval-augmented QA model in two specific scenarios: 1) adapting to updated versions of the same knowledge corpus; 2) switching to completely different knowledge domains. We observe that the generalization challenges of OpenQA models stem from the reader's over-reliance on memorizing the knowledge from the external corpus, which hinders the model from generalizing to a new knowledge corpus. We introduce Corpus-Invariant Tuning (CIT), a simple but effective training strategy, to mitigate the knowledge over-memorization by controlling the likelihood of retrieved contexts during training. Extensive experimental results on multiple OpenQA benchmarks show that CIT achieves significantly better generalizability without compromising the model's performance in its original corpus and domain.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 37 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

27 of 37 references.

References · 27

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  7. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  8. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  9. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  10. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  11. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  12. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 476
15 more
  1. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  2. 2022RealTime QA: What's the Answer Right Now?cited by 244
  3. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  4. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 225
  5. 2020Answering Complex Open-Domain Questions with Multi-Hop Dense Retrievalcited by 162
  6. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  7. 2023Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-Incited by 71
  8. 2022Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformercited by 45
  9. 2023RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Modelscited by 37
  10. 2020A Memory Efficient Baseline for Open Domain Question Answeringcited by 32
  11. 2021Retrieval-guided Counterfactual Generation for QAcited by 29
  12. 2022Open-Domain Conversational Question Answering with Historical Answerscited by 8
  13. 2022Synthetic Target Domain Supervision for Open Retrieval QAcited by 6
  14. 2020Knowledge-Aided Open-Domain Question Answeringcited by 5
  15. 2022Entity-Conditioned Question Generation for Robust Attention Distribution in Neural Information Retrievalcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Improving Retrieval Augmented Open-Domain Question-Answering with Vectorized Contexts
  2. 2022Augmenting Pre-trained Language Models with QA-Memory for Open-Domain Question Answering
  3. 2021End-to-End Training of Neural Retrievers for Open-Domain Question Answering
  4. 2024RetrievalQA: Assessing Adaptive Retrieval-Augmented Generation for Short-form Open-Domain Question Answering
  5. 2022Better Retrieval May Not Lead to Better Question Answering
  6. 2024RE-RAG: Improving Open-Domain QA Performance and Interpretability with Relevance Estimator in Retrieval-Augmented Generation
  7. 2024DEXTER: A Benchmark for open-domain Complex Question Answering using LLMs
  8. 2022Variational Open-Domain Question Answering