mlsanityData to 13 Oct 2026

Open Set Classification of GAN-based Image Manipulations via a ViT-based Hybrid Architecture

arXiv 2304.05212 ↗First posted 11 Apr 2023cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Classification of AI-manipulated content is receiving great attention, for distinguishing different types of manipulations. Most of the methods developed so far fail in the open-set scenario, that is when the algorithm used for the manipulation is not represented by the training set. In this paper, we focus on the classification of synthetic face generation and manipulation in open-set scenarios, and propose a method for classification with a rejection option. The proposed method combines the use of Vision Transformers (ViT) with a hybrid approach for simultaneous classification and localization. Feature map correlation is exploited by the ViT module, while a localization branch is employed as an attention mechanism to force the model to learn per-class discriminative features associated with the forgery when the manipulation is performed locally in the image. Rejection is performed by considering several strategies and analyzing the model output layers. The effectiveness of the proposed method is assessed for the task of classification of facial attribute editing and GAN attribution.

Key facts. Cited by 6 papers in the map (data through 13 Oct 2026). Builds on 35 of its 46 references that are in the map. First posted on arXiv 11 Apr 2023.

Builds on

35 of 46 references.

References · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  4. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,585
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,237
  6. 2020Energy-based Out-of-distribution Detectioncited by 1,297
  7. 2021Alias-Free Generative Adversarial Networkscited by 1,183
  8. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  9. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 692
  10. 2020A Survey on Visual Transformercited by 654
  11. 2018BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentationcited by 645
  12. 2021Score-based Generative Modeling in Latent Spacecited by 635
23 more
  1. 2015Towards Open Set Deep Networkscited by 535
  2. 2021Open-Set Recognition: a Good Closed-Set Classifier is All You Need?cited by 353
  3. 2021Multi-attentional Deepfake Detectioncited by 340
  4. 2018Recent Advances in Open Set Recognition: A Surveycited by 310
  5. 2019Detecting and Simulating Artifacts in GAN Fake Imagescited by 293
  6. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  7. 2020A Review on Generative Adversarial Networks: Algorithms, Theory, and Applicationscited by 209
  8. 2018Do GANs leave artificial fingerprints?cited by 190
  9. 2021Adversarial Reciprocal Points Learning for Open Set Recognitioncited by 153
  10. 2018Classification-Reconstruction Learning for Open-Set Recognitioncited by 146
  11. 2019C2AE: Class Conditioned Auto-Encoder for Open-set Recognitioncited by 142
  12. 2021Are GAN generated images easy to detect? A critical analysis of the state-of-the-artcited by 139
  13. 2018Robust Classification with Convolutional Prototype Learningcited by 123
  14. 2020Learning Open Set Network with Discriminative Reciprocal Pointscited by 115
  15. 2021M2TR: Multi-modal Multi-scale Transformers for Deepfake Detectioncited by 114
  16. 2021Towards Discovery and Attribution of Open-world GAN Generated Imagescited by 66
  17. 2022Deepfake Network Architecture Attributioncited by 47
  18. 2020FakeLocator: Robust Localization of GAN-Based Face Manipulationscited by 40
  19. 2022GAN-generated Faces Detection: A Survey and New Perspectivescited by 30
  20. 2022Class-Specific Semantic Reconstruction for Open Set Recognitioncited by 28
  21. 2021Detection and Localization of Facial Expression Manipulationscited by 15
  22. 2022Open Set Recognition using Vision Transformer with an Additional Detection Headcited by 2
  23. 2022An Architecture for the detection of GAN-generated Flood Images with Localization Capabilitiescited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 1; 2026: 1.

6 papers.

  1. 2024Synthetic Image Verification in the Era of Generative AI: What Works and What Isn't There Yetcited by 13
  2. 2024BOSC: A Backdoor-based Framework for Open Set Synthetic Image Attributioncited by 6
  3. 2024Continuous fake media detection: adapting deepfake detectors to new generative techniquescited by 6
  4. 2023A Siamese-based Verification System for Open-set Architecture Attribution of Synthetic Imagescited by 0
  5. 2025Open Set Face Forgery Detection via Dual-Level Evidence Collectioncited by 0
  6. 2026Unknown Aware AI-Generated Content Attributioncited by 0