# Attacks by Content: Automated Fact-checking is an AI Security Issue

arXiv: https://arxiv.org/abs/2510.11238 · HTML: https://mlsanity.com/paper/2510.11238/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 104 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Builds on (most cited first, 30 of 42)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs](https://mlsanity.com/paper/2401.06373/) (2024)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration](https://mlsanity.com/paper/1802.08802/) (2018)
- [MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims](https://mlsanity.com/paper/1909.03242/) (2019)
- [FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information](https://mlsanity.com/paper/2106.05707/) (2021)
- [AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web](https://mlsanity.com/paper/2305.13117/) (2023)
- [The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversation](https://mlsanity.com/paper/2312.09085/) (2023)
- [COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic](https://mlsanity.com/paper/2106.03794/) (2021)
- [That is a Known Lie: Detecting Previously Fact-Checked Claims](https://mlsanity.com/paper/2005.06058/) (2020)
- [Predicting Factuality of Reporting and Bias of News Media Sources](https://mlsanity.com/paper/1810.01765/) (2018)
- [The Limitations of Stylometry for Detecting Machine-Generated Fake News](https://mlsanity.com/paper/1908.09805/) (2019)
- [Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering](https://mlsanity.com/paper/2410.15999/) (2024)
- [Towards Automated Factchecking: Developing an Annotation Schema and Benchmark for Consistent Automated Claim Detection](https://mlsanity.com/paper/1809.08193/) (2018)
- [Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise](https://mlsanity.com/paper/2305.01579/) (2023)
- [Multimodal Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2305.13507/) (2023)
- [Synthetic Disinformation Attacks on Automated Fact Verification Systems](https://mlsanity.com/paper/2202.09381/) (2022)
- [What Was Written vs. Who Read It: News Media Profiling Using Text Analysis and Social Media Context](https://mlsanity.com/paper/2005.04518/) (2020)

## Built on by (most cited first, 4 of 4)

- [Latent Causal Void: Explicit Missing-Context Reconstruction for Misinformation Detection](https://mlsanity.com/paper/2605.12156/) (2026)
- [MEDIAREF: A Public Knowledge Store for Media Background Checks](https://mlsanity.com/paper/2607.02383/) (2026)
- [PIPES: Securing Agent Perception with Provenance and Priors](https://mlsanity.com/paper/2608.12789/) (2026)
- [Lazy Grounding: Attacking Search Agents with Factual Evidence](https://mlsanity.com/paper/2608.30303/) (2026)

## Abstract (arXiv)

When AI agents retrieve and reason over external documents, adversaries can manipulate the data they receive to subvert their behaviour. Previous research has studied indirect prompt injection, where the attacker injects malicious instructions. We argue that injection of instructions is not necessary to manipulate agents - attackers could instead supply biased, misleading, or false information. We term this an attack by content. Existing defenses, which focus on detecting hidden commands, are ineffective against attacks by content. To defend themselves and their users, agents must critically evaluate retrieved information, corroborating claims with external evidence and evaluating source trustworthiness. We argue that this is analogous to an existing NLP task, automated fact-checking, which we propose to repurpose as a cognitive self-defense tool for agents.