# Hierarchical Fine-Grained Image Forgery Detection and Localization

arXiv: https://arxiv.org/abs/2303.17111 · HTML: https://mlsanity.com/paper/2303.17111/

## Key facts

- Cited by 134 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 83 references that are in the map.
- First posted on arXiv 30 Mar 2023.

## Builds on (most cited first, 30 of 59)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [CNN-generated images are surprisingly easy to spot... for now](https://mlsanity.com/paper/1912.11035/) (2019)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Face X-ray for More General Face Forgery Detection](https://mlsanity.com/paper/1912.13458/) (2019)
- [Detecting and Simulating Artifacts in GAN Fake Images](https://mlsanity.com/paper/1907.06515/) (2019)
- [On the detection of synthetic images generated by diffusion models](https://mlsanity.com/paper/2211.00680/) (2022)
- [What makes fake images detectable? Understanding properties that generalize](https://mlsanity.com/paper/2008.10588/) (2020)
- [On the Detection of Digital Face Manipulation](https://mlsanity.com/paper/1910.01717/) (2019)

## Built on by (most cited first, 30 of 134)

- [FakeShield: Explainable Image Forgery Detection and Localization via Multi-modal Large Language Models](https://mlsanity.com/paper/2410.02761/) (2024)
- [EditGuard: Versatile Image Watermarking for Tamper Localization and Copyright Protection](https://mlsanity.com/paper/2312.08883/) (2023)
- [SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Model](https://mlsanity.com/paper/2412.04292/) (2024)
- [DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmark](https://mlsanity.com/paper/2405.19707/) (2024)
- [Detecting Multimedia Generated by Large AI Models: A Survey](https://mlsanity.com/paper/2402.00045/) (2024)
- [Deepfake Generation and Detection: A Benchmark and Survey](https://mlsanity.com/paper/2403.17881/) (2024)
- [Common Sense Reasoning for Deepfake Detection](https://mlsanity.com/paper/2402.00126/) (2024)
- [LEGION: Learning to Ground and Explain for Synthetic Image Detection](https://mlsanity.com/paper/2503.15264/) (2025)
- [AntifakePrompt: Prompt-Tuned Vision-Language Models are Fake Image Detectors](https://mlsanity.com/paper/2310.17419/) (2023)
- [Generalizable Synthetic Image Detection via Language-guided Contrastive Learning](https://mlsanity.com/paper/2305.13800/) (2023)
- [Detecting Images Generated by Deep Diffusion Models using their Local Intrinsic Dimensionality](https://mlsanity.com/paper/2307.02347/) (2023)
- [IML-ViT: Benchmarking Image Manipulation Localization by Vision Transformer](https://mlsanity.com/paper/2307.14863/) (2023)
- [IMDL-BenCo: A Comprehensive Benchmark and Codebase for Image Manipulation Detection & Localization](https://mlsanity.com/paper/2406.10580/) (2024)
- [Diffusion Facial Forgery Detection](https://mlsanity.com/paper/2401.15859/) (2024)
- [On Learning Multi-Modal Forgery Representation for Diffusion Generated Video Detection](https://mlsanity.com/paper/2410.23623/) (2024)
- [Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector](https://mlsanity.com/paper/2503.20188/) (2025)
- [Unlearnable Examples for Diffusion Models: Protect Data from Unauthorized Exploitation](https://mlsanity.com/paper/2306.01902/) (2023)
- [OmniGuard: Hybrid Manipulation Localization via Augmented Versatile Deep Image Watermarking](https://mlsanity.com/paper/2412.01615/) (2024)
- [Tame a Wild Camera: In-the-Wild Monocular Camera Calibration](https://mlsanity.com/paper/2306.10988/) (2023)
- [VideoShield: Regulating Diffusion-based Video Generation Models via Watermarking](https://mlsanity.com/paper/2501.14195/) (2025)
- [WildFake: A Large-scale Challenging Dataset for AI-Generated Images Detection](https://mlsanity.com/paper/2402.11843/) (2024)
- [Weakly-supervised deepfake localization in diffusion-generated images](https://mlsanity.com/paper/2311.04584/) (2023)
- [ChatGPT-Powered Hierarchical Comparisons for Image Classification](https://mlsanity.com/paper/2311.00206/) (2023)
- [ForensicHub: A Unified Benchmark & Codebase for All-Domain Fake Image Detection and Localization](https://mlsanity.com/paper/2505.11003/) (2025)
- [Detect Any Deepfakes: Segment Anything Meets Face Forgery Detection and Localization](https://mlsanity.com/paper/2306.17075/) (2023)
- [Bi-LORA: A Vision-Language Approach for Synthetic Image Detection](https://mlsanity.com/paper/2404.01959/) (2024)
- [Toward Generalizable Forgery Detection and Reasoning](https://mlsanity.com/paper/2503.21210/) (2025)
- [GIM: A Million-scale Benchmark for Generative Image Manipulation Detection and Localization](https://mlsanity.com/paper/2406.16531/) (2024)
- [DeCLIP: Decoding CLIP representations for deepfake localization](https://mlsanity.com/paper/2409.08849/) (2024)
- [Learning Human-Perceived Fakeness in AI-Generated Videos via Multimodal LLMs](https://mlsanity.com/paper/2509.22646/) (2025)

## Abstract (arXiv)

Differences in forgery attributes of images generated in CNN-synthesized and image-editing domains are large, and such differences make a unified image forgery detection and localization (IFDL) challenging. To this end, we present a hierarchical fine-grained formulation for IFDL representation learning. Specifically, we first represent forgery attributes of a manipulated image with multiple labels at different levels. Then we perform fine-grained classification at these levels using the hierarchical dependency between them. As a result, the algorithm is encouraged to learn both comprehensive features and inherent hierarchical nature of different forgery attributes, thereby improving the IFDL representation. Our proposed IFDL framework contains three components: multi-branch feature extractor, localization and classification modules. Each branch of the feature extractor learns to classify forgery attributes at one level, while localization and classification modules segment the pixel-level forgery region and detect image-level forgery, respectively. Lastly, we construct a hierarchical fine-grained dataset to facilitate our study. We demonstrate the effectiveness of our method on 7 different benchmarks, for both tasks of IFDL and forgery attribute classification. Our source code and dataset can be found: hrefhttps://github.com/CHELSEA234/HiFiIFDLgithub.com/CHELSEA234/HiFi-IFDL.