# How well are open sourced AI-generated image detection models out-of-the-box: A comprehensive benchmark study

arXiv: https://arxiv.org/abs/2602.07814 · HTML: https://mlsanity.com/paper/2602.07814/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 52 references that are in the map.
- First posted on arXiv 8 Feb 2026.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Domain Generalization: A Survey](https://mlsanity.com/paper/2103.02503/) (2021)
- [Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues](https://mlsanity.com/paper/2007.09355/) (2020)
- [GenImage: A Million-Scale Benchmark for Detecting AI-Generated Image](https://mlsanity.com/paper/2306.08571/) (2023)
- [WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection](https://mlsanity.com/paper/2101.01456/) (2021)
- [PatchCraft: Exploring Texture Patch for Efficient AI-generated Image Detection](https://mlsanity.com/paper/2311.12397/) (2023)
- [Any-Resolution AI-Generated Image Detection by Spectral Learning](https://mlsanity.com/paper/2411.19417/) (2024)
- [Deepfake Detection: A Comprehensive Survey from the Reliability Perspective](https://mlsanity.com/paper/2211.10881/) (2022)
- [Segment anything, from space?](https://mlsanity.com/paper/2304.13000/) (2023)
- [On Improving Cross-dataset Generalization of Deepfake Detectors](https://mlsanity.com/paper/2204.04285/) (2022)
- [One Detector to Rule Them All: Towards a General Deepfake Attack Detection Framework](https://mlsanity.com/paper/2105.00187/) (2021)
- [Towards Solving the DeepFake Problem : An Analysis on Improving DeepFake Detection using Dynamic Face Augmentation](https://mlsanity.com/paper/2102.09603/) (2021)
- [Continual Learning in Neural Networks](https://mlsanity.com/paper/1910.02718/) (2019)
- [Deepfake Forensics via An Adversarial Game](https://mlsanity.com/paper/2103.13567/) (2021)
- [Towards Benchmarking and Evaluating Deepfake Detection](https://mlsanity.com/paper/2203.02115/) (2022)
- [SoK: Systematization and Benchmarking of Deepfake Detectors in a Unified Framework](https://mlsanity.com/paper/2401.04364/) (2024)
- [Can Multi-modal (reasoning) LLMs work as deepfake detectors?](https://mlsanity.com/paper/2503.20084/) (2025)
- [Do Deepfake Detectors Work in Reality?](https://mlsanity.com/paper/2502.10920/) (2025)
- [LOTA: Bit-Planes Guided AI-Generated Image Detection](https://mlsanity.com/paper/2510.14230/) (2025)
- [A Timely Survey on Vision Transformer for Deepfake Detection](https://mlsanity.com/paper/2405.08463/) (2024)
- [AI-GenBench: A New Ongoing Benchmark for AI-Generated Image Detection](https://mlsanity.com/paper/2504.20865/) (2025)
- [Automated Extraction of Energy Systems Information from Remotely Sensed Data: A Review and Analysis](https://mlsanity.com/paper/2202.12939/) (2022)
- [Data-Driven Fairness Generalization for Deepfake Detection](https://mlsanity.com/paper/2412.16428/) (2024)
- [Fit for Purpose? Deepfake Detection in the Real World](https://mlsanity.com/paper/2510.16556/) (2025)
- [A New Approach to Improve Learning-based Deepfake Detection in Realistic Conditions](https://mlsanity.com/paper/2203.11807/) (2022)
- [Deepfake Detection: A Comparative Analysis](https://mlsanity.com/paper/2308.03471/) (2023)
- [CAMME: Adaptive Deepfake Image Detection with Multi-Modal Cross-Attention](https://mlsanity.com/paper/2505.18035/) (2025)

## Built on by (most cited first, 15 of 15)

- [GPT-Image-2 in the Wild: A Twitter Dataset of Self-Reported AI-Generated Images from the First Week of Deployment](https://mlsanity.com/paper/2604.25370/) (2026)
- [AIForge-Doc: A Benchmark for Detecting AI-Forged Tampering in Financial and Form Documents](https://mlsanity.com/paper/2602.20569/) (2026)
- [DOCFORGE-BENCH: A Comprehensive 0-shot Benchmark for Document Forgery Detection and Analysis](https://mlsanity.com/paper/2603.01433/) (2026)
- [GPT4o-Receipt: A Dataset and Human Study for AI-Generated Document Forensics](https://mlsanity.com/paper/2603.11442/) (2026)
- [Anatomy of a Scam Call: What 10,000 real scam and spam calls reveal about how phone scammers operate](https://mlsanity.com/paper/2608.24127/) (2026)
- [Can a Teenager Fool an AI? Evaluating Low-Cost Cosmetic Attacks on Age Estimation Systems](https://mlsanity.com/paper/2602.19539/) (2026)
- [A Synthetic Eye Movement Dataset for Script Reading Detection: Real Trajectory Replay on a 3D Simulator](https://mlsanity.com/paper/2604.05475/) (2026)
- [Abductive Corroboration of Probabilistic AI Models for Forensic Synthetic Media Detection](https://mlsanity.com/paper/2607.05434/) (2026)
- [CallScreenBench: Benchmarking Small Language Models as Phone Secretaries](https://mlsanity.com/paper/2608.01033/) (2026)
- [PatchHead: Learning Spatial Patch Evidence for Generalizable AI-Generated Image Detection](https://mlsanity.com/paper/2608.09223/) (2026)
- [ChatGPT Images 2.5 in the Wild: A Launch-Period Dataset and Detector Evaluation](https://mlsanity.com/paper/2609.15100/) (2026)
- [Benchmarking Neural Defend ARCAS 1B: A Foundational Multimodal Deepfake Detection Model](https://mlsanity.com/paper/2609.25154/) (2026)
- [A Corpus of Real Scam- and Spam-Call Conversations from an Active Voice-Agent Honeypot](https://mlsanity.com/paper/2609.29528/) (2026)
- [REALIS: A Curated Dataset for Studying the Challenges of AI Image Detection](https://mlsanity.com/paper/2609.32734/) (2026)
- [Trustworthy synthetic visual media: Evidence across the media lifecycle](https://mlsanity.com/paper/2609.34232/) (2026)

## Abstract (arXiv)

As AI-generated images proliferate across digital platforms, reliable detection methods have become critical for combating misinformation and maintaining content authenticity. While numerous deepfake detection methods have been proposed, existing benchmarks predominantly evaluate fine-tuned models, leaving a critical gap in understanding out-of-the-box performance -- the most common deployment scenario for practitioners. We present the first comprehensive zero-shot evaluation of 16 state-of-the-art detection methods, comprising 23 pretrained detector variants (due to multiple released versions of certain detectors), across 12 diverse datasets, comprising 2.6~million image samples spanning 291 unique generators including modern diffusion models. Our systematic analysis reveals striking findings: (1)~no universal winner exists, with detector rankings exhibiting substantial instability (Spearman~ρ: 0.01 -- 0.87 across dataset pairs); (2)~a 37~percentage-point performance gap separates the best detector (75.0% mean accuracy) from the worst (37.5%); (3)~training data alignment critically impacts generalization, causing up to 20--60% performance variance within architecturally identical detector families; (4)~modern commercial generators (Flux~Dev, Firefly~v4, Midjourney~v7) defeat most detectors, achieving only 18--30% average accuracy; and (5)~we identify three systematic failure patterns affecting cross-dataset generalization. Statistical analysis confirms significant performance differences between detectors (Friedman test: chi2=121.01, p<10-16, Kendall~W=0.524). Our findings challenge the ``one-size-fits-all'' detector paradigm and provide actionable deployment guidelines, demonstrating that practitioners must carefully select detectors based on their specific threat landscape rather than relying on published benchmark performance.