# Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributions

arXiv: https://arxiv.org/abs/2003.01826 · HTML: https://mlsanity.com/paper/2003.01826/

## Key facts

- Cited by 280 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2020.

## Built on by (most cited first, 30 of 280)

- [GenImage: A Million-Scale Benchmark for Detecting AI-Generated Image](https://mlsanity.com/paper/2306.08571/) (2023)
- [Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain](https://mlsanity.com/paper/2103.01856/) (2021)
- [Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detection](https://mlsanity.com/paper/2312.16649/) (2023)
- [ResViT: Residual vision transformers for multi-modal medical image synthesis](https://mlsanity.com/paper/2106.16031/) (2021)
- [DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection](https://mlsanity.com/paper/2307.01426/) (2023)
- [Raising the Bar of AI-generated Image Detection with CLIP](https://mlsanity.com/paper/2312.00195/) (2023)
- [Exploring Temporal Coherence for More General Video Face Forgery Detection](https://mlsanity.com/paper/2108.06693/) (2021)
- [Are GAN generated images easy to detect? A critical analysis of the state-of-the-art](https://mlsanity.com/paper/2104.02617/) (2021)
- [Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection](https://mlsanity.com/paper/2201.07131/) (2022)
- [Pros and Cons of GAN Evaluation Measures: New Developments](https://mlsanity.com/paper/2103.09396/) (2021)
- [Intriguing properties of synthetic images: from generative adversarial networks to diffusion models](https://mlsanity.com/paper/2304.06408/) (2023)
- [PatchCraft: Exploring Texture Patch for Efficient AI-generated Image Detection](https://mlsanity.com/paper/2311.12397/) (2023)
- [Towards the Detection of Diffusion Model Deepfakes](https://mlsanity.com/paper/2210.14571/) (2022)
- [Face Forensics in the Wild](https://mlsanity.com/paper/2103.16076/) (2021)
- [Self-Supervised Video Forensics by Audio-Visual Anomaly Detection](https://mlsanity.com/paper/2301.01767/) (2023)
- [Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forward](https://mlsanity.com/paper/2103.00484/) (2021)
- [WaveFake: A Data Set to Facilitate Audio Deepfake Detection](https://mlsanity.com/paper/2111.02813/) (2021)
- [FrePGAN: Robust Deepfake Detection Using Frequency-level Perturbations](https://mlsanity.com/paper/2202.03347/) (2022)
- [A Closer Look at Fourier Spectrum Discrepancies for CNN-generated Images Detection](https://mlsanity.com/paper/2103.17195/) (2021)
- [A Bias-Free Training Paradigm for More General AI-generated Image Detection](https://mlsanity.com/paper/2412.17671/) (2024)
- [BiHPF: Bilateral High-Pass Filters for Robust Deepfake Detection](https://mlsanity.com/paper/2109.00911/) (2021)
- [On the Frequency Bias of Generative Models](https://mlsanity.com/paper/2111.02447/) (2021)
- [Any-Resolution AI-Generated Image Detection by Spectral Learning](https://mlsanity.com/paper/2411.19417/) (2024)
- [Countering Malicious DeepFakes: Survey, Battleground, and Horizon](https://mlsanity.com/paper/2103.00218/) (2021)
- [WOUAF: Weight Modulation for User Attribution and Fingerprinting in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2306.04744/) (2023)
- [Real-Time Deepfake Detection in the Real-World](https://mlsanity.com/paper/2406.09398/) (2024)
- [Beyond the Prior Forgery Knowledge: Mining Critical Clues for General Face Forgery Detection](https://mlsanity.com/paper/2304.12489/) (2023)
- [Zero-Shot Detection of AI-Generated Images](https://mlsanity.com/paper/2409.15875/) (2024)
- [Community Forensics: Using Thousands of Generators to Train Fake Image Detectors](https://mlsanity.com/paper/2411.04125/) (2024)
- [Generalizable Synthetic Image Detection via Language-guided Contrastive Learning](https://mlsanity.com/paper/2305.13800/) (2023)

## Abstract (arXiv)

Generative convolutional deep neural networks, e.g. popular GAN architectures, are relying on convolution based up-sampling methods to produce non-scalar outputs like images or video sequences. In this paper, we show that common up-sampling methods, i.e. known as up-convolution or transposed convolution, are causing the inability of such models to reproduce spectral distributions of natural training data correctly. This effect is independent of the underlying architecture and we show that it can be used to easily detect generated data like deepfakes with up to 100% accuracy on public benchmarks. To overcome this drawback of current generative models, we propose to add a novel spectral regularization term to the training optimization objective. We show that this approach not only allows to train spectral consistent GANs that are avoiding high frequency errors. Also, we show that a correct approximation of the frequency spectrum has positive effects on the training stability and output quality of generative networks.