# Defending Against Disinformation Attacks in Open-Domain Question Answering

arXiv: https://arxiv.org/abs/2212.10002 · HTML: https://mlsanity.com/paper/2212.10002/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 20 Dec 2022.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering](https://mlsanity.com/paper/2012.00955/) (2020)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [Selective Question Answering under Domain Shift](https://mlsanity.com/paper/2006.09462/) (2020)
- [SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine](https://mlsanity.com/paper/1704.05179/) (2017)
- [Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence](https://mlsanity.com/paper/2210.13701/) (2022)
- [Simple and Effective Multi-Paragraph Reading Comprehension](https://mlsanity.com/paper/1710.10723/) (2017)
- [Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attacks](https://mlsanity.com/paper/2006.12557/) (2020)
- [Imitation Attacks and Defenses for Black-box Machine Translation Systems](https://mlsanity.com/paper/2004.15015/) (2020)
- [Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation](https://mlsanity.com/paper/2104.08678/) (2021)
- [Synthetic Disinformation Attacks on Automated Fact Verification Systems](https://mlsanity.com/paper/2202.09381/) (2022)
- [Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval](https://mlsanity.com/paper/2106.11251/) (2021)
- [Domain-agnostic Question-Answering with Adversarial Training](https://mlsanity.com/paper/1910.09342/) (2019)

## Built on by (most cited first, 12 of 12)

- [Certifiably Robust RAG against Retrieval Corruption](https://mlsanity.com/paper/2405.15556/) (2024)
- [Trustworthiness in Retrieval-Augmented Generation Systems: A Survey](https://mlsanity.com/paper/2409.10102/) (2024)
- [Retrieval-Augmented Generation with Conflicting Evidence](https://mlsanity.com/paper/2504.13079/) (2025)
- [WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia](https://mlsanity.com/paper/2406.13805/) (2024)
- [CrAM: Credibility-Aware Attention Modification in LLMs for Combating Misinformation in RAG](https://mlsanity.com/paper/2406.11497/) (2024)
- [Retrieval-Augmented Generation with Estimation of Source Reliability](https://mlsanity.com/paper/2410.22954/) (2024)
- [OpenDecoder: Open Large Language Model Decoding to Incorporate Document Quality in RAG](https://mlsanity.com/paper/2601.09028/) (2026)
- [Exploring the Impact of Instruction-Tuning on LLM's Susceptibility to Misinformation](https://mlsanity.com/paper/2507.18203/) (2025)
- [Through the Stealth Lens: Attention-Aware Defenses Against Poisoning in RAG](https://mlsanity.com/paper/2506.04390/) (2025)
- [HealthContradict: Evaluating Biomedical Knowledge Conflicts in Language Models](https://mlsanity.com/paper/2512.02299/) (2025)
- [LLM-Oriented Information Retrieval: A Denoising-First Perspective](https://mlsanity.com/paper/2605.00505/) (2026)
- [Navigating Unreliable Parametric and Contextual Knowledge: Explicit Knowledge Conflict Resolution for LLM Inference](https://mlsanity.com/paper/2606.20245/) (2026)

## Abstract (arXiv)

Recent work in open-domain question answering (ODQA) has shown that adversarial poisoning of the search collection can cause large drops in accuracy for production systems. However, little to no work has proposed methods to defend against these attacks. To do so, we rely on the intuition that redundant information often exists in large corpora. To find it, we introduce a method that uses query augmentation to search for a diverse set of passages that could answer the original question but are less likely to have been poisoned. We integrate these new passages into the model through the design of a novel confidence method, comparing the predicted answer to its appearance in the retrieved contexts (what we call Confidence from Answer Redundancy, i.e. CAR). Together these methods allow for a simple but effective way to defend against poisoning attacks that provides gains of nearly 20% exact match across varying levels of data poisoning/knowledge conflicts.