# Bridging the Gap Between Ideal and Real-world Evaluation: Benchmarking AI-Generated Image Detection in Challenging Scenarios

arXiv: https://arxiv.org/abs/2509.09172 · HTML: https://mlsanity.com/paper/2509.09172/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 76 of its 105 references that are in the map.
- First posted on arXiv 11 Sep 2025.

## Builds on (most cited first, 30 of 76)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [CNN-generated images are surprisingly easy to spot... for now](https://mlsanity.com/paper/1912.11035/) (2019)
- [Leveraging Frequency Analysis for Deep Fake Image Recognition](https://mlsanity.com/paper/2003.08685/) (2020)
- [Towards Universal Fake Image Detectors that Generalize Across Generative Models](https://mlsanity.com/paper/2302.10174/) (2023)
- [DIRE for Diffusion-Generated Image Detection](https://mlsanity.com/paper/2303.09295/) (2023)
- [GenImage: A Million-Scale Benchmark for Detecting AI-Generated Image](https://mlsanity.com/paper/2306.08571/) (2023)
- [Detecting and Simulating Artifacts in GAN Fake Images](https://mlsanity.com/paper/1907.06515/) (2019)
- [On the detection of synthetic images generated by diffusion models](https://mlsanity.com/paper/2211.00680/) (2022)
- [Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributions](https://mlsanity.com/paper/2003.01826/) (2020)
- [What makes fake images detectable? Understanding properties that generalize](https://mlsanity.com/paper/2008.10588/) (2020)
- [Global Texture Enhancement for Fake Face Detection in the Wild](https://mlsanity.com/paper/2002.00133/) (2020)
- [A Sanity Check for AI-generated Image Detection](https://mlsanity.com/paper/2406.19435/) (2024)
- [Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detection](https://mlsanity.com/paper/2312.16649/) (2023)
- [DE-FAKE: Detection and Attribution of Fake Images Generated by Text-to-Image Generation Models](https://mlsanity.com/paper/2210.06998/) (2022)
- [Raising the Bar of AI-generated Image Detection with CLIP](https://mlsanity.com/paper/2312.00195/) (2023)
- [Fourier Spectrum Discrepancies in Deep Network Generated Images](https://mlsanity.com/paper/1911.06465/) (2019)
- [AEROBLADE: Training-Free Detection of Latent Diffusion Images Using Autoencoder Reconstruction Error](https://mlsanity.com/paper/2401.17879/) (2024)

## Built on by (most cited first, 24 of 24)

- [NTIRE 2026 Challenge on Robust AI-Generated Image Detection in the Wild](https://mlsanity.com/paper/2604.11487/) (2026)
- [Simplicity Prevails: The Emergence of Generalizable AIGI Detection in Visual Foundation Models](https://mlsanity.com/paper/2602.01738/) (2026)
- [TreeTeaming: Autonomous Red-Teaming of Vision-Language Models via Hierarchical Strategy Exploration](https://mlsanity.com/paper/2603.22882/) (2026)
- [AgentFoX: LLM Agent-Guided Fusion with eXplainability for AI-Generated Image Detection](https://mlsanity.com/paper/2603.23115/) (2026)
- [Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models](https://mlsanity.com/paper/2604.14629/) (2026)
- [Quality-Aware Calibration for AI-Generated Image Detection in the Wild](https://mlsanity.com/paper/2604.15027/) (2026)
- [DailyBench: A Unified Benchmark for AI-Generated and Manipulated Images from Modern Generative Models](https://mlsanity.com/paper/2607.24016/) (2026)
- [Veritas++: Value-aware On-Policy Distillation for Perception-Enhanced AIGI Detection](https://mlsanity.com/paper/2607.27113/) (2026)
- [A Benchmark Dataset for MLLM-Generated Image Detection: GPT Image2 & Nano Banana2](https://mlsanity.com/paper/2608.01258/) (2026)
- [TranX-Adapter: Bridging Artifacts and Semantics within MLLMs for Robust AI-generated Image Detection](https://mlsanity.com/paper/2602.21716/) (2026)
- [HEDGE: Heterogeneous Ensemble for Detection of AI-GEnerated Images in the Wild](https://mlsanity.com/paper/2604.03555/) (2026)
- [Video as Natural Augmentation: Towards Unified AI-Generated Image and Video Detection](https://mlsanity.com/paper/2605.21977/) (2026)
- [Dissect and Prune: Enhancing Robustness in AI-Generated Image Detection](https://mlsanity.com/paper/2606.10309/) (2026)
- [SalArt-VQA: Diagnosing Whether VLMs Understand Salient Artifacts in Generated Images](https://mlsanity.com/paper/2606.12671/) (2026)
- [GlobalForge: Towards Robust AI-Generated Image Detection](https://mlsanity.com/paper/2607.14684/) (2026)
- [Automated In-the-Wild Data Collection for Continual AI Generated Image Detection](https://mlsanity.com/paper/2605.02567/) (2026)
- [SPECTRA-Net: Scalable Pipeline for Explainable Cross-domain Tensor Representations for AI-generated Images Detection](https://mlsanity.com/paper/2605.08226/) (2026)
- [Structured Local Differential Modeling for AI-Generated Image Detection](https://mlsanity.com/paper/2608.12811/) (2026)
- [Prior-Conditioned Gaussian Discriminants for Generalizable AI-generated Image Detection](https://mlsanity.com/paper/2608.18523/) (2026)
- [Learning Continuous Source Responses For Generalizable AI-Generated Image Detection](https://mlsanity.com/paper/2609.14316/) (2026)
- [ChatGPT Images 2.5 in the Wild: A Launch-Period Dataset and Detector Evaluation](https://mlsanity.com/paper/2609.15100/) (2026)
- [REALIS: A Curated Dataset for Studying the Challenges of AI Image Detection](https://mlsanity.com/paper/2609.32734/) (2026)
- [MAD-Guard: Controlled Study of Autoregressive Generation versus Direct Decision Interfaces for Closed Multimodal Forensic Tasks](https://mlsanity.com/paper/2609.33683/) (2026)
- [Trustworthy synthetic visual media: Evidence across the media lifecycle](https://mlsanity.com/paper/2609.34232/) (2026)

## Abstract (arXiv)

With the rapid advancement of generative models, highly realistic image synthesis has posed new challenges to digital security and media credibility. Although AI-generated image detection methods have partially addressed these concerns, a substantial research gap remains in evaluating their performance under complex real-world conditions. This paper introduces the Real-World Robustness Dataset (RRDataset) for comprehensive evaluation of detection models across three dimensions: 1) Scenario Generalization: RRDataset encompasses high-quality images from seven major scenarios (War and Conflict, Disasters and Accidents, Political and Social Events, Medical and Public Health, Culture and Religion, Labor and Production, and everyday life), addressing existing dataset gaps from a content perspective. 2) Internet Transmission Robustness: examining detector performance on images that have undergone multiple rounds of sharing across various social media platforms. 3) Re-digitization Robustness: assessing model effectiveness on images altered through four distinct re-digitization methods. We benchmarked 17 detectors and 10 vision-language models (VLMs) on RRDataset and conducted a large-scale human study involving 192 participants to investigate human few-shot learning capabilities in detecting AI-generated images. The benchmarking results reveal the limitations of current AI detection methods under real-world conditions and underscore the importance of drawing on human adaptability to develop more robust detection algorithms.