When the Forger Is the Judge: GPT-Image-2 Cannot Recognize Its Own Faked Documents
Not on the map. Why?
Abstract (arXiv)
OpenAI's GPT-Image-2 has effectively erased the visual boundary between authentic and AI-edited document images: a single number on a receipt can be replaced in under a second for a few cents. We release AIForge-Doc v2, a paired dataset of 3,066 GPT-Image-2 document forgeries with pixel-precise masks in DocTamper-compatible format, and benchmark four lines of defence: human inspectors (N=120, n=365 pair-votes via the public 2AFC site CanUSpotAI.com), TruFor (generic forensic), DocTamper (qcf-568, document-specific), and the same GPT-Image-2 model as a zero-shot self-judge -- asked, to avoid the trivial "image is mostly real" reading, whether any region was generated or edited by an AI image model. Human 2AFC accuracy is 0.501, indistinguishable from chance: even side-by-side, inspectors cannot tell GPT-Image-2 receipt forgeries from authentic counterparts. The three computational judges sit only modestly above (TruFor 0.599, DocTamper 0.585, self-judge 0.532). The self-judge fails consistently, not by chance: across five prompt strategies and four policies for handling ambiguous responses, AUC never rises above 0.59. To rule out the possibility that the two forensic detectors are broken on our source domain rather than blind to AI inpainting, we calibrate each on a same-domain traditional-tampering set built for its training distribution: TruFor reaches AUC 0.962 on cross-camera splicing of our dataset, DocTamper reaches 0.852 on cross-document OCR-token splicing with two-pass JPEG re-encoding. Both retain near-published performance on traditional tampering; switching to GPT-Image-2 inpainting drops AUC by 0.27-0.36 (0.962->0.599 TruFor; 0.852->0.585 DocTamper), isolating a detection gap specific to GPT-Image-2 inpainting. We release the dataset, pipeline, four-judge protocol, and calibration sets.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 13 of its 24 references that are in the map. First posted on arXiv 28 Apr 2026.
Builds on
13 of 24 references.
References · 13
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
- 2023Hierarchical Fine-Grained Image Forgery Detection and Localizationcited by 134
- 2021PSCC-Net: Progressive Spatio-Channel Correlation Network for Image Manipulation Detection and Localizationcited by 133
- 2022TruFor: Leveraging all-round clues for trustworthy image forgery detection and localizationcited by 130
- 2024AEROBLADE: Training-Free Detection of Latent Diffusion Images Using Autoencoder Reconstruction Errorcited by 123
- 2021Spatial Dual-Modality Graph Reasoning for Key Information Extractioncited by 43
- 2023IML-ViT: Benchmarking Image Manipulation Localization by Vision Transformercited by 42
- 2024Revisiting Tampered Scene Text Detection in the Era of Generative AIcited by 21
- 2023Toward Real Text Manipulation Detection: New Dataset and New Solutioncited by 20
- 2025Can Multi-modal (reasoning) LLMs work as deepfake detectors?cited by 14
- 2025SAGI: Semantically Aligned and Uncertainty Guided AI Image Inpaintingcited by 10
- 2025Can Multi-modal (reasoning) LLMs detect document manipulation?cited by 7
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026ChatGPT Images 2.5 on Forgery Tasks: Testing Advertised Improvements Against Known Answerscited by 2
- 2026SynCred-Bench: Benchmarking Synthetic Credibility in AI-Generated Visual Misinformationcited by 1
- 2026Agents That Edit Documents: Measuring Agentic PDF Forgery Against a Non-Agentic Controlcited by 0
- 2026Agents That Edit Documents: Measuring Agentic PDF Forgery Against a Non-Agentic Control20 Sep 2026
- 2026ChatGPT Images 2.5 on Forgery Tasks: Testing Advertised Improvements Against Known Answers12 Sep 2026
- 2026SynCred-Bench: Benchmarking Synthetic Credibility in AI-Generated Visual Misinformation2 Jun 2026