# Distributed Denial of Science: How Indirect Data Poisoning of AI Systems Can Industrialize Scientific Fraud arXiv: https://arxiv.org/abs/2607.10712 ยท HTML: https://mlsanity.com/paper/2607.10712/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. - Builds on 35 of its 91 references that are in the map. - First posted on arXiv 12 Jul 2026. ## Placed in - LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/) ## Builds on (most cited first, 30 of 35) - [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014) - [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023) - [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017) - [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://mlsanity.com/paper/2506.13131/) (2025) - [Agent Laboratory: Using LLM Agents as Research Assistants](https://mlsanity.com/paper/2501.04227/) (2025) - [Accelerating scientific discovery with Co-Scientist](https://mlsanity.com/paper/2502.18864/) (2025) - [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://mlsanity.com/paper/2310.18018/) (2023) - [Poisoning Web-Scale Training Datasets is Practical](https://mlsanity.com/paper/2302.10149/) (2023) - [PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models](https://mlsanity.com/paper/2402.07867/) (2024) - [Towards End-to-End Automation of AI Research](https://mlsanity.com/paper/2606.15497/) (2026) - [Benchmark Data Contamination of Large Language Models: A Survey](https://mlsanity.com/paper/2406.04244/) (2024) - [From Automation to Autonomy: A Survey on Large Language Models in Scientific Discovery](https://mlsanity.com/paper/2505.13259/) (2025) - [Authenticated Delegation and Authorized AI Agents](https://mlsanity.com/paper/2501.09674/) (2025) - [LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions](https://mlsanity.com/paper/2411.05025/) (2024) - [Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025](https://mlsanity.com/paper/2504.09737/) (2025) - [Practical Poisoning Attacks against Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.03957/) (2025) - [Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agents](https://mlsanity.com/paper/2502.16069/) (2025) - [Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?](https://mlsanity.com/paper/2502.14297/) (2025) - [The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems](https://mlsanity.com/paper/2509.08713/) (2025) - [SciSciGPT: Advancing Human-AI Collaboration in the Science of Science](https://mlsanity.com/paper/2504.05559/) (2025) - [Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks](https://mlsanity.com/paper/2512.03262/) (2025) - [AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot](https://mlsanity.com/paper/2604.13940/) (2026) - [CPA-RAG:Covert Poisoning Attacks on Retrieval-Augmented Generation in Large Language Models](https://mlsanity.com/paper/2505.19864/) (2025) - [AI scientists produce results without reasoning scientifically](https://mlsanity.com/paper/2604.18805/) (2026) - [Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment](https://mlsanity.com/paper/2411.03417/) (2024) - [The Adoption and Usage of AI Agents: Early Evidence from Perplexity](https://mlsanity.com/paper/2512.07828/) (2025) - [Compound Deception in Elite Peer Review: A Failure Mode Taxonomy of 100 Fabricated Citations at NeurIPS 2025](https://mlsanity.com/paper/2602.05930/) (2026) - [Exploring the use of AI authors and reviewers at Agents4Science](https://mlsanity.com/paper/2511.15534/) (2025) - [Why LLMs Aren't Scientists Yet: Lessons from Four Autonomous Research Attempts](https://mlsanity.com/paper/2601.03315/) (2026) - [Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generation](https://mlsanity.com/paper/2503.06254/) (2025) ## Built on by (most cited first, 2 of 2) - [PIPES: Securing Agent Perception with Provenance and Priors](https://mlsanity.com/paper/2608.12789/) (2026) - [AI in Science: Early Insights](https://mlsanity.com/paper/2609.28504/) (2026) ## Abstract (arXiv) Scientific fraud is the instrument of doubt that malicious entities can use to establish controversy in science. Historically, it required the resources of a company: deep pockets, ghostwritten articles, and corrupt academics. Today, Artificial Intelligence (AI) is increasingly automating scientific research, so we ask: Can a remote adversary weaponize the honest use of AI in science to compromise scientific integrity? We envision and empirically evaluate a new attack, indirect data poisoning, in which an adversary corrupts an open dataset and uploads the poisoned variant to a public repository. Autonomous research agents may independently retrieve and process this data, turning honest scientists into the unpaid and unwitting distributors of fraud at scale. Across five socially-salient topics, from hiring discrimination to the safety of autonomous vehicles, three widely used frontier AI systems (Claude Code with Claude Opus 4.7, Codex with GPT-5.5, Gemini CLI with Gemini 3.1 Pro), and 450 ethically contained experimental runs, we find that poisoning succeeds in 49.56% of runs, while the rate of poisoning detection is only 6.0%. The attack requires no topic-specific trigger-words, agent access, indirect prompt injection, or fabricated papers, only the open data ecosystem and misleading metadata. To mitigate the attacks, we propose and evaluate two measures: a scientist persona and a data provenance audit with five checks (referencing papers, social markers, statistical anomalies, related datasets, poisoning caution). We find that the persona still leaves 16.67% of runs with a poisoned conclusion, but provenance auditing reduces attack success rate to zero. Our results suggest that indirect data poisoning may enable scientific fraud at unprecedented scale, but these attacks can be mitigated with suitable auditing by agents during data retrieval.