# Mandela-Bench: Multimodal Models Remember Canonical Images Instead of Seeing Them

arXiv: https://arxiv.org/abs/2609.32763 · HTML: https://mlsanity.com/paper/2609.32763/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 45 of its 58 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Builds on (most cited first, 30 of 45)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [Towards Universal Fake Image Detectors that Generalize Across Generative Models](https://mlsanity.com/paper/2302.10174/) (2023)
- [UltraEdit: Instruction-based Fine-Grained Image Editing at Scale](https://mlsanity.com/paper/2407.05282/) (2024)
- [Gemma 4 Technical Report](https://mlsanity.com/paper/2607.02770/) (2026)
- [A Sanity Check for AI-generated Image Detection](https://mlsanity.com/paper/2406.19435/) (2024)
- [MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs](https://mlsanity.com/paper/2502.17422/) (2025)
- [Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning](https://mlsanity.com/paper/2503.15558/) (2025)
- [Kimi K3: Open Frontier Intelligence](https://mlsanity.com/paper/2607.24653/) (2026)
- [MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction](https://mlsanity.com/paper/2604.27393/) (2026)
- [Deepfake Detection by Human Crowds, Machines, and Machine-informed Crowds](https://mlsanity.com/paper/2105.06496/) (2021)
- [Detecting and Grounding Multi-Modal Media Manipulation and Beyond](https://mlsanity.com/paper/2309.14203/) (2023)
- [LOKI: A Comprehensive Synthetic Data Detection Benchmark using Large Multimodal Models](https://mlsanity.com/paper/2410.09732/) (2024)
- [Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circulated in 2024](https://mlsanity.com/paper/2503.02857/) (2025)
- [Rethinking the Up-Sampling Operations in CNN-based Generative Network for Generalizable Deepfake Detection](https://mlsanity.com/paper/2312.10461/) (2023)
- [FakeBench: Probing Explainable Fake Image Detection via Large Multimodal Models](https://mlsanity.com/paper/2404.13306/) (2024)
- [Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts](https://mlsanity.com/paper/2505.17127/) (2025)
- [Celeb-DF++: A Large-scale Challenging Video DeepFake Benchmark for Generalizable Forensics](https://mlsanity.com/paper/2507.18015/) (2025)
- [NTIRE 2026 Challenge on Robust AI-Generated Image Detection in the Wild](https://mlsanity.com/paper/2604.11487/) (2026)
- [MiRAGeNews: Multimodal Realistic AI-Generated News Detection](https://mlsanity.com/paper/2410.09045/) (2024)
- [Navigating the Challenges of AI-Generated Image Detection in the Wild: What Truly Matters?](https://mlsanity.com/paper/2507.10236/) (2025)

## Abstract (arXiv)

Historical photographs and other canonical images can now be edited seamlessly with a single instruction, often leaving no reliable pixel-level trace. In such cases, the only evidence of manipulation may be a fact about what the image depicts. Existing benchmarks instead rely on generator artefacts, image-caption inconsistencies, visual implausibilities, or external references, and therefore do not test whether a model can use its own world knowledge to verify a recognized image. We introduce Mandela-Bench, containing 1,507 edits of canonical images: 1,359 knowledge-only forgeries, each contradicting one verifiable fact, and 148 anchor-free controls that preserve the editing process without introducing a factual contradiction, together with 474 untouched originals. We score not only whether a model detects a forgery, but whether its explanation identifies the inserted entity or the fact being violated. Across 36 multimodal models, from 0.8B parameters to frontier scale, we find a consistent failure mode. When a public figure is removed from a familiar photograph, models still name that person in up to 72.7% of responses. Some models can distinguish the replacement face from the original when shown in isolation, yet still judge the full edited photograph as authentic. Providing the true event and date does not improve knowledge-grounded detection, whereas providing the same information after cropping away the recognizable composition does. Even under explicit verification prompts, only one of the 36 models meets the KGR criterion on at least half of the forged images. These results suggest that the failures cannot be explained by missing knowledge or inadequate perception alone. Instead, they are consistent with recognition biasing verification toward the remembered canonical image rather than the observed edit.