mlsanityData to 7 Oct 2026

Attacking Open-domain Question Answering by Injecting Misinformation

arXiv 2110.07803 ↗First posted 15 Oct 2021cs.CL cs.AIcited by 41

Not on the map. Why?

Abstract (arXiv)

With a rise in false, inaccurate, and misleading information in propaganda, news, and social media, real-world Question Answering (QA) systems face the challenges of synthesizing and reasoning over misinformation-polluted contexts to derive correct answers. This urgency gives rise to the need to make QA systems robust to misinformation, a topic previously unexplored. We study the risk of misinformation to QA models by investigating the sensitivity of open-domain QA models to corpus pollution with misinformation documents. We curate both human-written and model-generated false documents that we inject into the evidence corpus of QA models and assess the impact on the performance of these systems. Experiments show that QA models are vulnerable to even small amounts of evidence contamination brought by misinformation, with large absolute performance drops on all models. Misinformation attack brings more threat when fake documents are produced at scale by neural models or the attacker targets hacking specific questions of interest. To defend against such a threat, we discuss the necessity of building a misinformation-aware QA system that integrates question-answering and misinformation detection in a joint fashion.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 31 of its 47 references that are in the map. First posted on arXiv 15 Oct 2021.

Builds on

31 of 47 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  11. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  12. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
19 more
  1. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  2. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  3. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  4. 2019Defending Against Neural Fake Newscited by 553
  5. 2021A Survey on Automated Fact-Checkingcited by 376
  6. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  7. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  8. 2020Selective Question Answering under Domain Shiftcited by 228
  9. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 212
  10. 2017Learning to Ask: Neural Question Generation for Reading Comprehensioncited by 186
  11. 2019End-to-End Open-Domain Question Answering with BERTserinicited by 152
  12. 2021An Empirical Survey of Data Augmentation for Limited Data Learning in NLPcited by 101
  13. 2018How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarkscited by 86
  14. 2019Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QAcited by 63
  15. 2018Did the Model Understand the Question?cited by 60
  16. 2021Efficient Passage Retrieval with Hashing for Open-domain Question Answeringcited by 53
  17. 2019Generating Sentiment-Preserving Fake Online Reviews Using Neural Language Models and Their Human- and Machine-based Detectioncited by 50
  18. 2021Zero-shot Fact Verification by Claim Generationcited by 42
  19. 2019Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generationcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 18; 2026: 12.

41 papers.

Sort:

By area:

  1. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  2. 2024Certifiably Robust RAG against Retrieval Corruptioncited by 144
  3. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  4. 2024What Evidence Do Language Models Find Convincing?cited by 74
  5. 2025SealQA: Raising the Bar for Reasoning in Search-Augmented Language Modelscited by 65
  6. 2024Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineeringcited by 60
  7. 2025Retrieval-Augmented Generation with Conflicting Evidencecited by 60
  8. 2024How Susceptible are LLMs to Influence in Prompts?cited by 32
  9. 2025SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of Large Language Modelcited by 31
  10. 2025RbFT: Robust Fine-tuning for Retrieval-Augmented Generation against Retrieval Defectscited by 27
  11. 2025MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attackscited by 24
  12. 2024CrAM: Credibility-Aware Attention Modification in LLMs for Combating Misinformation in RAGcited by 22
29 more
  1. 2024Open Domain Question Answering with Conflicting Contextscited by 18
  2. 2025The RAG Paradox: A Black-Box Attack Exploiting Unintentional Vulnerabilities in Retrieval-Augmented Generation Systemscited by 14
  3. 2023Evaluating Large Language Models for Health-related Queries with Presuppositionscited by 12
  4. 2024Human-inspired Perspectives: A Survey on AI Long-term Memorycited by 12
  5. 2025Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarkingcited by 11
  6. 2024Toward Robust RALMs: Revealing the Impact of Imperfect Retrieval on Retrieval-Augmented Language Modelscited by 8
  7. 2026Whose Facts Win? LLM Source Preferences under Knowledge Conflictscited by 8
  8. 2025Unraveling Misinformation Propagation in LLM Reasoningcited by 6
  9. 2025MedFact: A Large-scale Chinese Dataset for Evidence-based Medical Fact-checking of LLM Responsescited by 6
  10. 2025Exploiting Instruction-Following Retrievers for Malicious Information Retrievalcited by 4
  11. 2025Exploring the Impact of Instruction-Tuning on LLM's Susceptibility to Misinformationcited by 4
  12. 2025Through the Stealth Lens: Attention-Aware Defenses Against Poisoning in RAGcited by 3
  13. 2025Bridging External and Parametric Knowledge: Mitigating Hallucination of LLMs with Shared-Private Semantic Synergy in Dual-Stream Knowledgecited by 3
  14. 2025Conflicts in Texts: Data, Implications and Challengescited by 2
  15. 2025When Facts Change: Probing LLMs on Evolving Knowledge with evolveQAcited by 2
  16. 2026Understanding the Dynamics of Demonstration Conflict in In-Context Learningcited by 2
  17. 2026MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environmentscited by 2
  18. 2025Harnessing the Unseen: The Hidden Influence of Intrinsic Knowledge in Long-Context Language Modelscited by 1
  19. 2025MillStone: How Open-Minded Are LLMs?cited by 1
  20. 2025Out-of-Context Misinformation Detection via Variational Domain-Invariant Learning with Test-Time Trainingcited by 1
  21. 2026Safer Policy Compliance with Dynamic Epistemic Fallbackcited by 1
  22. 2026REAL: Resolving Knowledge Conflicts in Knowledge-Intensive Visual Question Answering via Reasoning-Pivot Alignmentcited by 1
  23. 2026Benchmarking Vision-Language Models under Contradictory Virtual Content Attacks in Augmented Realitycited by 1
  24. 2026LLM-Oriented Information Retrieval: A Denoising-First Perspectivecited by 1
  25. 2026Navigating Unreliable Parametric and Contextual Knowledge: Explicit Knowledge Conflict Resolution for LLM Inferencecited by 1
  26. 2026Information Discernment in Large Language Modelscited by 1
  27. 2026Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generationcited by 0
  28. 2026ConflictScore: Identifying and Measuring How Language Models Handle Conflicting Evidencecited by 0
  29. 2026Large Language Models in Misinformation Ecosystems: Misuse, Defense, and Vulnerabilitycited by 0