Distributed Denial of Science: How Indirect Data Poisoning of AI Systems Can Industrialize Scientific Fraud
On the map: LLM safety and security → Backdoor and data poisoning attacks
Abstract (arXiv)
Scientific fraud is the instrument of doubt that malicious entities can use to establish controversy in science. Historically, it required the resources of a company: deep pockets, ghostwritten articles, and corrupt academics. Today, Artificial Intelligence (AI) is increasingly automating scientific research, so we ask: Can a remote adversary weaponize the honest use of AI in science to compromise scientific integrity? We envision and empirically evaluate a new attack, indirect data poisoning, in which an adversary corrupts an open dataset and uploads the poisoned variant to a public repository. Autonomous research agents may independently retrieve and process this data, turning honest scientists into the unpaid and unwitting distributors of fraud at scale. Across five socially-salient topics, from hiring discrimination to the safety of autonomous vehicles, three widely used frontier AI systems (Claude Code with Claude Opus 4.7, Codex with GPT-5.5, Gemini CLI with Gemini 3.1 Pro), and 450 ethically contained experimental runs, we find that poisoning succeeds in 49.56% of runs, while the rate of poisoning detection is only 6.0%. The attack requires no topic-specific trigger-words, agent access, indirect prompt injection, or fabricated papers, only the open data ecosystem and misleading metadata. To mitigate the attacks, we propose and evaluate two measures: a scientist persona and a data provenance audit with five checks (referencing papers, social markers, statistical anomalies, related datasets, poisoning caution). We find that the persona still leaves 16.67% of runs with a poisoned conclusion, but provenance auditing reduces attack success rate to zero. Our results suggest that indirect data poisoning may enable scientific fraud at unprecedented scale, but these attacks can be mitigated with suitable auditing by agents during data retrieval.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 35 of its 91 references that are in the map. First posted on arXiv 12 Jul 2026.
Builds on
35 of 91 references.
References · 35
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
- 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,106
- 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 867
- 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 425
- 2025Accelerating scientific discovery with Co-Scientistcited by 401
- 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
- 2023Poisoning Web-Scale Training Datasets is Practicalcited by 290
- 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 267
- 2026Towards End-to-End Automation of AI Researchcited by 159
- 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
- 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 93
23 more
- 2025Authenticated Delegation and Authorized AI Agentscited by 65
- 2024LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptionscited by 47
- 2025Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025cited by 47
- 2025Practical Poisoning Attacks against Retrieval-Augmented Generationcited by 36
- 2025Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agentscited by 33
- 2025Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?cited by 32
- 2025The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systemscited by 29
- 2025SciSciGPT: Advancing Human-AI Collaboration in the Science of Sciencecited by 27
- 2025Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Taskscited by 22
- 2026AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilotcited by 19
- 2025CPA-RAG:Covert Poisoning Attacks on Retrieval-Augmented Generation in Large Language Modelscited by 17
- 2026AI scientists produce results without reasoning scientificallycited by 17
- 2024Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experimentcited by 16
- 2025The Adoption and Usage of AI Agents: Early Evidence from Perplexitycited by 15
- 2026Compound Deception in Elite Peer Review: A Failure Mode Taxonomy of 100 Fabricated Citations at NeurIPS 2025cited by 15
- 2025Exploring the use of AI authors and reviewers at Agents4Sciencecited by 13
- 2026Why LLMs Aren't Scientists Yet: Lessons from Four Autonomous Research Attemptscited by 13
- 2025Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generationcited by 12
- 2025Defending Against Knowledge Poisoning Attacks During Retrieval-Augmented Generationcited by 11
- 2023Causal Explanations for Sequential Decision-Making in Multi-Agent Systemscited by 10
- 2026LLM hallucinations in the wild: Large-scale evidence from non-existent citationscited by 7
- 2026Scaling Reproducibility: An AI-Assisted Workflow for Large-Scale Replication and Reanalysiscited by 4
- 2025Integrating Counterfactual Simulations with Language Models for Explaining Multi-Agent Behaviourcited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.
- 2026PIPES: Securing Agent Perception with Provenance and Priorscited by 0
- 2026AI in Science: Early Insightscited by 0
- 2026AI in Science: Early Insights15 Sep 2026
- 2026PIPES: Securing Agent Perception with Provenance and Priors13 Aug 2026
Similar, not cited
Close in wording, no citation either way
- 2026A Case Study on Emergent Cheating and Whistleblowing in Autonomous Research Swarms
- 2026Oracle Poisoning: Corrupting Knowledge Graphs to Weaponise AI Agent Reasoning
- 2026Poisoned Playbooks: Demystifying Knowledge Poisoning Effects on AI Security Agents
- 2026The Age of AI Agents Demands A New Scientific Paradigm To Sustain Trustworthy Science
- 2026PseudoBench: Measuring How Agentic Auto-Research Fuels Pseudoscience
- 2026Malicious Repurposing of Open Science Artefacts by Using Large Language Models
- 2025Attacks by Content: Automated Fact-checking is an AI Security Issue
- 2025Detecting and Preventing Data Poisoning Attacks on AI Models