# Backbone is All You Need: Assessing Vulnerabilities of Frozen Foundation Models in Synthetic Image Forensics

arXiv: https://arxiv.org/abs/2605.13381 · HTML: https://mlsanity.com/paper/2605.13381/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 13 May 2026.

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [On the detection of synthetic images generated by diffusion models](https://mlsanity.com/paper/2211.00680/) (2022)
- [DE-FAKE: Detection and Attribution of Fake Images Generated by Text-to-Image Generation Models](https://mlsanity.com/paper/2210.06998/) (2022)
- [Raising the Bar of AI-generated Image Detection with CLIP](https://mlsanity.com/paper/2312.00195/) (2023)
- [More Real than Real: A Study on Human Visual Perception of Synthetic Faces](https://mlsanity.com/paper/2106.07226/) (2021)
- [Seeing What Matters: Generalizable AI-generated Video Detection with Forensic-Oriented Augmentation](https://mlsanity.com/paper/2506.16802/) (2025)
- [SoK: Systematization and Benchmarking of Deepfake Detectors in a Unified Framework](https://mlsanity.com/paper/2401.04364/) (2024)
- [TrueFake: A Real World Case Dataset of Last Generation Fake Images also Shared on Social Networks](https://mlsanity.com/paper/2504.20658/) (2025)
- [When Synthetic Traces Hide Real Content: Analysis of Stable Diffusion Image Laundering](https://mlsanity.com/paper/2407.10736/) (2024)
- [Conditioned Prompt-Optimization for Continual Deepfake Detection](https://mlsanity.com/paper/2407.21554/) (2024)
- [MF-CLIP: Leveraging CLIP as Surrogate Models for No-box Adversarial Attacks](https://mlsanity.com/paper/2307.06608/) (2023)
- [Advance Fake Video Detection via Vision Transformers](https://mlsanity.com/paper/2504.20669/) (2025)

## Abstract (arXiv)

As AI-generated synthetic images become increasingly realistic, Vision Transformers (ViTs) have emerged as a cornerstone of modern deepfake detection. However, the prevailing reliance on frozen, pre-trained backbones introduces a subtle yet critical vulnerability. In this work, we present the Surrogate Iterative Adversarial Attack (SIAA), a gray-box attack that exploits knowledge of the detector's ViT backbone alone and operates entirely within the target detector's feature space to craft highly effective adversarial examples. Through our experiments, involving multiple ViT-based detectors and diverse gray-box scenarios, including few-shot learning, complete training misalignment and attack transferability tests, we demonstrate that this vulnerability consistently yields high attack success rates, often approaching white-box performance. By doing so, we reveal that backbone knowledge alone is sufficient to undermine detector reliability, highlighting the urgent need for more resilient defenses in adversarial multimedia forensics.