# Fake or JPEG? Revealing Common Biases in Generated Image Detection Datasets

arXiv: https://arxiv.org/abs/2403.17608 · HTML: https://mlsanity.com/paper/2403.17608/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 30 references that are in the map.
- First posted on arXiv 26 Mar 2024.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [CNN-generated images are surprisingly easy to spot... for now](https://mlsanity.com/paper/1912.11035/) (2019)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [Towards Universal Fake Image Detectors that Generalize Across Generative Models](https://mlsanity.com/paper/2302.10174/) (2023)
- [DIRE for Diffusion-Generated Image Detection](https://mlsanity.com/paper/2303.09295/) (2023)
- [GenImage: A Million-Scale Benchmark for Detecting AI-Generated Image](https://mlsanity.com/paper/2306.08571/) (2023)
- [Detecting and Simulating Artifacts in GAN Fake Images](https://mlsanity.com/paper/1907.06515/) (2019)
- [Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributions](https://mlsanity.com/paper/2003.01826/) (2020)
- [Are GAN generated images easy to detect? A critical analysis of the state-of-the-art](https://mlsanity.com/paper/2104.02617/) (2021)
- [Intriguing properties of synthetic images: from generative adversarial networks to diffusion models](https://mlsanity.com/paper/2304.06408/) (2023)
- [Towards the Detection of Diffusion Model Deepfakes](https://mlsanity.com/paper/2210.14571/) (2022)
- [Seeing is not always believing: Benchmarking Human and Model Perception of AI-Generated Images](https://mlsanity.com/paper/2304.13023/) (2023)
- [Shadows Don't Lie and Lines Can't Bend! Generative Models don't know Projective Geometry...for now](https://mlsanity.com/paper/2311.17138/) (2023)
- [Online Detection of AI-Generated Images](https://mlsanity.com/paper/2310.15150/) (2023)
- [WOUAF: Weight Modulation for User Attribution and Fingerprinting in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2306.04744/) (2023)
- [GenDet: Towards Good Generalizations for AI-Generated Image Detection](https://mlsanity.com/paper/2312.08880/) (2023)
- [Deep Image Fingerprint: Towards Low Budget Synthetic Image Detection and Model Lineage Analysis](https://mlsanity.com/paper/2303.10762/) (2023)
- [Towards Universal GAN Image Detection](https://mlsanity.com/paper/2112.12606/) (2021)

## Built on by (most cited first, 30 of 43)

- [A Sanity Check for AI-generated Image Detection](https://mlsanity.com/paper/2406.19435/) (2024)
- [A Bias-Free Training Paradigm for More General AI-generated Image Detection](https://mlsanity.com/paper/2412.17671/) (2024)
- [Dual Data Alignment Makes AI-Generated Image Detector Easier Generalizable](https://mlsanity.com/paper/2505.14359/) (2025)
- [Real-Time Deepfake Detection in the Real-World](https://mlsanity.com/paper/2406.09398/) (2024)
- [Zero-Shot Detection of AI-Generated Images](https://mlsanity.com/paper/2409.15875/) (2024)
- [Community Forensics: Using Thousands of Generators to Train Fake Image Detectors](https://mlsanity.com/paper/2411.04125/) (2024)
- [CO-SPY: Combining Semantic and Pixel Features to Detect Synthetic Images by AI](https://mlsanity.com/paper/2503.18286/) (2025)
- [Aligned Datasets Improve Detection of Latent Diffusion-Generated Images](https://mlsanity.com/paper/2410.11835/) (2024)
- [Seeing What Matters: Generalizable AI-generated Video Detection with Forensic-Oriented Augmentation](https://mlsanity.com/paper/2506.16802/) (2025)
- [Your One-Stop Solution for AI-Generated Video Detection](https://mlsanity.com/paper/2601.11035/) (2026)
- [Stay-Positive: A Case for Ignoring Real Image Features in Fake Image Detection](https://mlsanity.com/paper/2502.07778/) (2025)
- [Towards Explainable Fake Image Detection with Multi-Modal Large Language Models](https://mlsanity.com/paper/2504.14245/) (2025)
- [Few-Shot Synthetic Image Attribution: Identifying Unseen Generators with Limited Samples](https://mlsanity.com/paper/2509.25682/) (2025)
- [Where Detectors Fail: Probing Generative Space for Generalizable AI-Generated Image Detection](https://mlsanity.com/paper/2605.24906/) (2026)
- [Fully AI-Generated Image Detection: Definition, Recent Advances and Challenges](https://mlsanity.com/paper/2502.19716/) (2025)
- [DRAGON: A Large-Scale Dataset of Realistic Images Generated by Diffusion Models](https://mlsanity.com/paper/2505.11257/) (2025)
- [Present and Future Generalization of Synthetic Image Detectors](https://mlsanity.com/paper/2409.14128/) (2024)
- [Processing and acquisition traces in visual encoders: What does CLIP know about your camera?](https://mlsanity.com/paper/2508.10637/) (2025)
- [Time Step Generating: A Universal Synthesized Deepfake Image Detector](https://mlsanity.com/paper/2411.11016/) (2024)
- [Rethinking the Use of Vision Transformers for AI-Generated Image Detection](https://mlsanity.com/paper/2512.04969/) (2025)
- [AI-Generated Image Detectors Overrely on Global Artifacts: Evidence from Inpainting Exchange](https://mlsanity.com/paper/2602.00192/) (2026)
- [How Fragile Are Training-Free AI-Generated Image Detectors? A Controlled Audit of Score Direction, Preprocessing, and Compression](https://mlsanity.com/paper/2606.20488/) (2026)
- [GAMMA: Generalizable Alignment via Multi-task and Manipulation-Augmented Training for AI-Generated Image Detection](https://mlsanity.com/paper/2509.10250/) (2025)
- [Deepfakes: we need to re-think the concept of "real" images](https://mlsanity.com/paper/2509.21864/) (2025)
- [EIRES:Training-free AI-Generated Image Detection via Edit-Induced Reconstruction Error Shift](https://mlsanity.com/paper/2510.25141/) (2025)
- [RA-Det: Towards Universal Detection of AI-Generated Images via Robustness Asymmetry](https://mlsanity.com/paper/2603.01544/) (2026)
- [Reduce the Artifacts Bias for More Generalizable AI-Generated Image Detection](https://mlsanity.com/paper/2605.14486/) (2026)
- [BIAS-ID: A Framework for Analyzing Transformation Biases in AI-Generated Image Detectors](https://mlsanity.com/paper/2605.31153/) (2026)
- [Dissect and Prune: Enhancing Robustness in AI-Generated Image Detection](https://mlsanity.com/paper/2606.10309/) (2026)
- [Dataset Biases and Shortcut Learning in Motion-Based AI-Generated Video Detection](https://mlsanity.com/paper/2607.00948/) (2026)

## Abstract (arXiv)

The widespread adoption of generative image models has highlighted the urgent need to detect artificial content, which is a crucial step in combating widespread manipulation and misinformation. Consequently, numerous detectors and associated datasets have emerged. However, many of these datasets inadvertently introduce undesirable biases, thereby impacting the effectiveness and evaluation of detectors. In this paper, we emphasize that many datasets for AI-generated image detection contain biases related to JPEG compression and image size. Using the GenImage dataset, we demonstrate that detectors indeed learn from these undesired factors. Furthermore, we show that removing the named biases substantially increases robustness to JPEG compression and significantly alters the cross-generator performance of evaluated detectors. Specifically, it leads to more than 11 percentage points increase in cross-generator performance for ResNet50 and Swin-T detectors on the GenImage dataset, achieving state-of-the-art results. We provide the dataset and source codes of this paper on the anonymous website: https://www.unbiased-genimage.org