# VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection

arXiv: https://arxiv.org/abs/2607.06254 · HTML: https://mlsanity.com/paper/2607.06254/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 43 of its 94 references that are in the map.
- First posted on arXiv 7 Jul 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 30 of 43)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [CNN-generated images are surprisingly easy to spot... for now](https://mlsanity.com/paper/1912.11035/) (2019)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics](https://mlsanity.com/paper/1909.12962/) (2019)
- [Leveraging Frequency Analysis for Deep Fake Image Recognition](https://mlsanity.com/paper/2003.08685/) (2020)
- [Towards Universal Fake Image Detectors that Generalize Across Generative Models](https://mlsanity.com/paper/2302.10174/) (2023)
- [DIRE for Diffusion-Generated Image Detection](https://mlsanity.com/paper/2303.09295/) (2023)
- [SelectiveNet: A Deep Neural Network with an Integrated Reject Option](https://mlsanity.com/paper/1901.09192/) (2019)
- [GenImage: A Million-Scale Benchmark for Detecting AI-Generated Image](https://mlsanity.com/paper/2306.08571/) (2023)
- [Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributions](https://mlsanity.com/paper/2003.01826/) (2020)
- [DeepFakes: a New Threat to Face Recognition? Assessment and Detection](https://mlsanity.com/paper/1812.08685/) (2018)
- [The Deepfake Detection Challenge (DFDC) Preview Dataset](https://mlsanity.com/paper/1910.08854/) (2019)
- [WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection](https://mlsanity.com/paper/2101.01456/) (2021)
- [A Sanity Check for AI-generated Image Detection](https://mlsanity.com/paper/2406.19435/) (2024)
- [Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detection](https://mlsanity.com/paper/2312.16649/) (2023)
- [DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection](https://mlsanity.com/paper/2307.01426/) (2023)
- [Raising the Bar of AI-generated Image Detection with CLIP](https://mlsanity.com/paper/2312.00195/) (2023)
- [DF40: Toward Next-Generation Deepfake Detection](https://mlsanity.com/paper/2406.13495/) (2024)
- [ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis](https://mlsanity.com/paper/2103.05630/) (2021)
- [AEROBLADE: Training-Free Detection of Latent Diffusion Images Using Autoencoder Reconstruction Error](https://mlsanity.com/paper/2401.17879/) (2024)
- [Fast Face-swap Using Convolutional Neural Networks](https://mlsanity.com/paper/1611.09577/) (2016)
- [FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces](https://mlsanity.com/paper/1803.09179/) (2018)
- [C2P-CLIP: Injecting Category Common Prompt in CLIP to Enhance Generalization in Deepfake Detection](https://mlsanity.com/paper/2408.09647/) (2024)

## Abstract (arXiv)

Deepfake image detection is served by three fundamentally different paradigms - commercial APIs, zero-shot vision-language models (LLMs), and open-source detectors - that are rarely evaluated under a common protocol, making direct comparison difficult. We introduce VendorBench-100, a cross-paradigm benchmark that evaluates 36 representative models using a single adversarial 100-image corpus, a unified output schema, and a common evaluation framework. Models are ranked primarily by the Matthews correlation coefficient (MCC), with ROC-AUC reported as a threshold-independent measure of ranking ability. Rather than maximizing size, it emphasizes real-world difficulty through a taxonomy of eight edge-case families such as face swaps, text-to-video stills, AI photo edits, avatar compositing, opaque-provenance images, and compressed research frames. Commercial APIs achieve the strongest median performance, followed by vision LLMs and open-source detectors, though individual open-source models remain competitive with the best LLMs. Across all 36 models, MCC and ROC-AUC are strongly correlated (Pearson r ~ 0.86); the more consequential finding is narrower and one-directional: a subset of otherwise strong rankers are miscalibrated at their shipped default threshold, so a high ROC-AUC can overstate real-world deployability. Separately, raw accuracy and F1 are unreliable on this corpus's imbalanced class split, since a model that predicts "fake" indiscriminately scores deceptively well on both while offering no real discriminative skill. No single metric is safe in isolation: MCC and specificity should always accompany ROC-AUC and accuracy. We release the complete evaluation framework and results. Code and data: https://github.com/sharayu-20/vendorbench-100