Agents That Edit Documents: Measuring Agentic PDF Forgery Against a Non-Agentic Control
On the map: LLM agents and tool use → Agent threats, guardrails and governance also fits: Financial trading and forecasting agents
Abstract (arXiv)
AI agents that carry a multi-step computer task through on their own became ordinary tools in the past year, and the same autonomy is available to anyone whose task is harmful. We ask what that means for a relying party -- an insurer, a lender, an auditor -- whose evidence is a filed PDF. AgentForge-Bench measures how reliably an off-the-shelf coding agent, driving one of seven open-weight models with a shell and the stock Python PDF stack, alters one dollar amount, date or address in a real filed financial document from a single sentence of intent, graded by rules rather than by a model. Across 1,750 cells, 1,419 (81.1%) satisfy the verifier, and 808 (46.2%) also survive every stricter filter: visible, localized, typeface-matched, original value gone document-wide. A deterministic script with no model in it solves 98 of the 125 documents; the agents solve 124, and none the script solves alone. Agents misreport 41% of their wrong edits as done, no model refused, and the cheapest verified forgery costs 2.4 cents. The raw rate overstates the threat by about a factor of two; the strict rate is still large.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 24 of its 29 references that are in the map. First posted on arXiv 20 Sep 2026.
Builds on
24 of 29 references.
References · 24
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
- 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 164
- 2023Hierarchical Fine-Grained Image Forgery Detection and Localizationcited by 134
- 2021PSCC-Net: Progressive Spatio-Channel Correlation Network for Image Manipulation Detection and Localizationcited by 133
- 2022TruFor: Leveraging all-round clues for trustworthy image forgery detection and localizationcited by 130
- 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 128
- 2021MVSS-Net: Multi-View Multi-Scale Supervised Networks for Image Manipulation Detectioncited by 117
- 2024FakeShield: Explainable Image Forgery Detection and Localization via Multi-modal Large Language Modelscited by 113
12 more
- 2024SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Modelcited by 108
- 2024IMDL-BenCo: A Comprehensive Benchmark and Codebase for Image Manipulation Detection & Localizationcited by 42
- 2025ForensicHub: A Unified Benchmark & Codebase for All-Domain Fake Image Detection and Localizationcited by 22
- 2024ForgeryGPT: A Multimodal LLM for Interpretable Image Forgery Detection and Localizationcited by 13
- 2024Omni-IML: Towards Unified Image Manipulation Localizationcited by 7
- 2025Can Multi-modal (reasoning) LLMs detect document manipulation?cited by 7
- 2026AIForge-Doc: A Benchmark for Detecting AI-Forged Tampering in Financial and Form Documentscited by 5
- 2025Weakly-Supervised Image Forgery Localization via Vision-Language Collaborative Reasoning Frameworkcited by 4
- 2026DOCFORGE-BENCH: A Comprehensive 0-shot Benchmark for Document Forgery Detection and Analysiscited by 4
- 2026GPT4o-Receipt: A Dataset and Human Study for AI-Generated Document Forensicscited by 4
- 2026When the Forger Is the Judge: GPT-Image-2 Cannot Recognize Its Own Faked Documentscited by 3
- 2026ChatGPT Images 2.5 on Forgery Tasks: Testing Advertised Improvements Against Known Answerscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Fidelity Is Not Enough: Dispatch-Level Instrumentation for Agentic Datasheet Extraction
- 2026LegalHalluLens: Typed Hallucination Auditing and Calibrated Multi-Agent Debate for Trustworthy Legal AI
- 2026Who Checks the Citations? Benchmarking Legal Hallucination Detection
- 2026Subjects, Not Authors: The Authorship Hazard in Agentic Dataspaces
- 2026Beyond the Text: Verifying That Agent-Written Papers Are Backed by Their Artifacts
- 2026Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI
- 2026Agentic Commerce Bench: Measuring Fraud Detection for Agents That Spend Money
- 2026ParseBench: A Document Parsing Benchmark for AI Agents