mlsanityData to 14 Oct 2026

On the Risk of Misinformation Pollution with Large Language Models

arXiv 2305.13661 ↗First posted 23 May 2023cs.CL cs.AIcited by 128

Not on the map. Why?

Cited by 23 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we comprehensively investigate the potential misuse of modern Large Language Models (LLMs) for generating credible-sounding misinformation and its subsequent impact on information-intensive applications, particularly Open-Domain Question Answering (ODQA) systems. We establish a threat model and simulate potential misuse scenarios, both unintentional and intentional, to assess the extent to which LLMs can be utilized to produce misinformation. Our study reveals that LLMs can act as effective misinformation generators, leading to a significant degradation in the performance of ODQA systems. To mitigate the harm caused by LLM-generated misinformation, we explore three defense strategies: prompting, misinformation detection, and majority voting. While initial results show promising trends for these defensive strategies, much more work needs to be done to address the challenge of misinformation pollution. Our work highlights the need for further research and interdisciplinary collaboration to address LLM-generated misinformation and to promote responsible use of LLMs.

Key facts. Cited by 128 papers in the map (data through 14 Oct 2026). Builds on 31 of its 46 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

31 of 46 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  11. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  12. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 726
19 more
  1. 2019Defending Against Neural Fake Newscited by 559
  2. 2023Can AI-Generated Text be Reliably Detected?cited by 386
  3. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
  4. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
  5. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 298
  6. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 290
  7. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  8. 2023Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigationscited by 158
  9. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 144
  10. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 127
  11. 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 126
  12. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  13. 2023On the Possibilities of AI-Generated Text Detectioncited by 88
  14. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 80
  15. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
  16. 2022Faking Fake News for Real Fake News Detection: Propaganda-loaded Training Data Generationcited by 41
  17. 2021Answering Open-Domain Multi-Answer Questions via a Recall-then-Verify Frameworkcited by 11
  18. 2022Who Funds Misinformation? A Systematic Analysis of the Ad-related Profit Routines of Fake News sitescited by 7
  19. 2022Context Generation Improves Open Domain Question Answeringcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 17; 2024: 43; 2025: 45; 2026: 23.

128 papers.

Sort:

By area:

  1. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
  2. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 267
  3. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  4. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  5. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 186
  6. 2023Can LLM-Generated Misinformation Be Detected?cited by 177
  7. 2024Certifiably Robust RAG against Retrieval Corruptioncited by 151
  8. 2023Factuality Challenges in the Era of Large Language Modelscited by 126
  9. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  10. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 103
  11. 2024FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"cited by 86
  12. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
38 more
  1. 2023Fake News in Sheep's Clothing: Robust Fake News Detection Against LLM-Empowered Style Attackscited by 70
  2. 2023An Unforgeable Publicly Verifiable Watermark for Large Language Modelscited by 65
  3. 2025Retrieval-Augmented Generation with Conflicting Evidencecited by 63
  4. 2024Raidar: geneRative AI Detection viA Rewritingcited by 52
  5. 2023Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrongcited by 51
  6. 2024Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts When Knowledge Conflicts?cited by 48
  7. 2024Steering Large Language Models using Conceptors: Improving Addition-Based Activation Engineeringcited by 39
  8. 2025Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistenciescited by 39
  9. 2023Neural Retrievers are Biased Towards LLM-Generated Contentcited by 38
  10. 2024Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanesecited by 36
  11. 2025GUARDIAN: Safeguarding LLM Multi-Agent Collaborations with Temporal Graph Modelingcited by 36
  12. 2024DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Modelscited by 34
  13. 2025PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimizationcited by 32
  14. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directionscited by 31
  15. 2023Trojan Activation Attack: Red-Teaming Large Language Models using Activation Steering for Safety-Alignmentcited by 30
  16. 2024Exploring the Deceptive Power of LLM-Generated Fake News: A Study of Real-World Detection Challengescited by 30
  17. 2024Farsight: Fostering Responsible AI Awareness During AI Application Prototypingcited by 28
  18. 2025RbFT: Robust Fine-tuning for Retrieval-Augmented Generation against Retrieval Defectscited by 27
  19. 2026"Humans welcome to observe": A First Look at the Agent Social Network Moltbookcited by 26
  20. 2024To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contextscited by 25
  21. 2025FlippedRAG: Black-Box Opinion Manipulation Adversarial Attacks to Retrieval-Augmented Generation Modelscited by 25
  22. 2023Zero-Shot Detection of Machine-Generated Codescited by 24
  23. 2024The Typing Cure: Experiences with Large Language Model Chatbots for Mental Health Supportcited by 24
  24. 2024Claim Verification in the Age of Large Language Models: A Surveycited by 24
  25. 2024Detecting AI-Generated Text: Factors Influencing Detectability with Current Methodscited by 23
  26. 2023Citation: A Key to Building Responsible and Accountable Large Language Modelscited by 20
  27. 2024Retrieval-Augmented Generation with Estimation of Source Reliabilitycited by 19
  28. 2023Fake News Detectors are Biased against Texts Generated by Large Language Modelscited by 18
  29. 2023User Modeling in the Era of Large Language Models: Current Research and Future Directionscited by 18
  30. 2024Loki: An Open-Source Tool for Fact Verificationcited by 16
  31. 2026Overcoming the Retrieval Barrier: Indirect Prompt Injection in the Wild for LLM Systemscited by 16
  32. 2023Adapting Fake News Detection to the Era of Large Language Modelscited by 14
  33. 2024Decoding the AI Pen: Techniques and Challenges in Detecting AI-Generated Textcited by 14
  34. 2024Has this Fact been Edited? Detecting Knowledge Edits in Language Modelscited by 14
  35. 2024Large Language Model Watermark Stealing With Mixed Integer Programmingcited by 14
  36. 2024On the (In)Security of LLM App Storescited by 14
  37. 2024Glimpse: Enabling White-Box Methods to Use Proprietary Models for Zero-Shot LLM-Generated Text Detectioncited by 14
  38. 2024Towards More Robust Retrieval-Augmented Generation: Evaluating RAG Under Adversarial Poisoning Attackscited by 14

50 most cited of 128 All 128 (JSON)