# Learning on Less: Constraining Pre-trained Model Learning for Generalizable Diffusion-Generated Image Detection

arXiv: https://arxiv.org/abs/2412.00665 · HTML: https://mlsanity.com/paper/2412.00665/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 18 of its 49 references that are in the map.
- First posted on arXiv 1 Dec 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Leveraging Frequency Analysis for Deep Fake Image Recognition](https://mlsanity.com/paper/2003.08685/) (2020)
- [GenImage: A Million-Scale Benchmark for Detecting AI-Generated Image](https://mlsanity.com/paper/2306.08571/) (2023)
- [Detecting GAN generated Fake Images using Co-occurrence Matrices](https://mlsanity.com/paper/1903.06836/) (2019)
- [ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection](https://mlsanity.com/paper/1812.02510/) (2018)
- [Leveraging Representations from Intermediate Encoder-blocks for Synthetic Image Detection](https://mlsanity.com/paper/2402.19091/) (2024)
- [Detecting GAN-generated Imagery using Color Cues](https://mlsanity.com/paper/1812.08247/) (2018)
- [Exposing the Fake: Effective Diffusion-Generated Images Detection](https://mlsanity.com/paper/2307.06272/) (2023)

## Built on by (most cited first, 2 of 2)

- [End4: End-to-end Denoising Diffusion for Diffusion-Based Inpainting Detection](https://mlsanity.com/paper/2509.13214/) (2025)
- [Your AI-Generated Image Detector Can Secretly Achieve SOTA Accuracy, If Calibrated](https://mlsanity.com/paper/2602.01973/) (2026)

## Abstract (arXiv)

Diffusion Models enable realistic image generation, raising the risk of misinformation and eroding public trust. Currently, detecting images generated by unseen diffusion models remains challenging due to the limited generalization capabilities of existing methods. To address this issue, we rethink the effectiveness of pre-trained models trained on large-scale, real-world images. Our findings indicate that: 1) Pre-trained models can cluster the features of real images effectively. 2) Models with pre-trained weights can approximate an optimal generalization solution at a specific training step, but it is extremely unstable. Based on these facts, we propose a simple yet effective training method called Learning on Less (LoL). LoL utilizes a random masking mechanism to constrain the model's learning of the unique patterns specific to a certain type of diffusion model, allowing it to focus on less image content. This leverages the inherent strengths of pre-trained weights while enabling a more stable approach to optimal generalization, which results in the extraction of a universal feature that differentiates various diffusion-generated images from real images. Extensive experiments on the GenImage benchmark demonstrate the remarkable generalization capability of our proposed LoL. With just 1% training data, LoL significantly outperforms the current state-of-the-art, achieving a 13.6% improvement in average ACC across images generated by eight different models.