mlsanityData to 7 Oct 2026

Mandela-Bench: Multimodal Models Remember Canonical Images Instead of Seeing Them

arXiv 2609.32763 ↗First posted 26 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Historical photographs and other canonical images can now be edited seamlessly with a single instruction, often leaving no reliable pixel-level trace. In such cases, the only evidence of manipulation may be a fact about what the image depicts. Existing benchmarks instead rely on generator artefacts, image-caption inconsistencies, visual implausibilities, or external references, and therefore do not test whether a model can use its own world knowledge to verify a recognized image. We introduce Mandela-Bench, containing 1,507 edits of canonical images: 1,359 knowledge-only forgeries, each contradicting one verifiable fact, and 148 anchor-free controls that preserve the editing process without introducing a factual contradiction, together with 474 untouched originals. We score not only whether a model detects a forgery, but whether its explanation identifies the inserted entity or the fact being violated. Across 36 multimodal models, from 0.8B parameters to frontier scale, we find a consistent failure mode. When a public figure is removed from a familiar photograph, models still name that person in up to 72.7% of responses. Some models can distinguish the replacement face from the original when shown in isolation, yet still judge the full edited photograph as authentic. Providing the true event and date does not improve knowledge-grounded detection, whereas providing the same information after cropping away the recognizable composition does. Even under explicit verification prompts, only one of the 36 models meets the KGR criterion on at least half of the forged images. These results suggest that the failures cannot be explained by missing knowledge or inadequate perception alone. Instead, they are consistent with recognition biasing verification toward the remembered canonical image rather than the observed edit.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 45 of its 58 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

45 of 58 references.

References · 45

  1. 2025Qwen2.5-VL Technical Reportcited by 5,418
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  4. 2025Qwen3-VL Technical Reportcited by 2,534
  5. 2025Gemma 3 Technical Reportcited by 1,686
  6. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  7. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  8. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  9. 2025Qwen2.5-Omni Technical Reportcited by 704
  10. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  11. 2023Towards Universal Fake Image Detectors that Generalize Across Generative Modelscited by 483
  12. 2025Qwen3-Omni Technical Reportcited by 451
33 more
  1. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 266
  2. 2024A Sanity Check for AI-generated Image Detectioncited by 187
  3. 2026Gemma 4 Technical Reportcited by 187
  4. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
  5. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 159
  6. 2026Kimi K3: Open Frontier Intelligencecited by 99
  7. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 64
  8. 2021Deepfake Detection by Human Crowds, Machines, and Machine-informed Crowdscited by 58
  9. 2023Detecting and Grounding Multi-Modal Media Manipulation and Beyondcited by 55
  10. 2024LOKI: A Comprehensive Synthetic Data Detection Benchmark using Large Multimodal Modelscited by 51
  11. 2023Rethinking the Up-Sampling Operations in CNN-based Generative Network for Generalizable Deepfake Detectioncited by 41
  12. 2025Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circulated in 2024cited by 40
  13. 2024FakeBench: Probing Explainable Fake Image Detection via Large Multimodal Modelscited by 36
  14. 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfactscited by 30
  15. 2025Celeb-DF++: A Large-scale Challenging Video DeepFake Benchmark for Generalizable Forensicscited by 18
  16. 2026NTIRE 2026 Challenge on Robust AI-Generated Image Detection in the Wildcited by 18
  17. 2024MiRAGeNews: Multimodal Realistic AI-Generated News Detectioncited by 15
  18. 2023A Template Is All You Memecited by 9
  19. 2025OpenFake: An Open Dataset and Platform Toward Real-World Deepfake Detectioncited by 9
  20. 2025Navigating the Challenges of AI-Generated Image Detection in the Wild: What Truly Matters?cited by 8
  21. 2025Semantic Visual Anomaly Detection and Reasoning in AI-Generated Imagescited by 7
  22. 2025ManipShield: A Unified Framework for Image Manipulation Detection, Localization and Explanationcited by 5
  23. 2026Pixels Don't Lie (But Your Detector Might): Bootstrapping MLLM-as-a-Judge for Trustworthy Deepfake Detection and Reasoning Supervisioncited by 5
  24. 2026VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learningcited by 5
  25. 2025Fully AI-Generated Image Detection: Definition, Recent Advances and Challengescited by 4
  26. 2024LookupForensics: A Large-Scale Multi-Task Dataset for Multi-Phase Image-Based Fact Verificationcited by 2
  27. 2026DailyBench: A Unified Benchmark for AI-Generated and Manipulated Images from Modern Generative Modelscited by 2
  28. 2025UniAIDet: A Unified and Universal Benchmark for AI-Generated Image Content Detection and Localizationcited by 1
  29. 2026Semantic Manipulation Localizationcited by 1
  30. 2026EditSleuth: A Dataset of Grounded Reasoning Chains for Image-Edit Forensicscited by 1
  31. 2026REVEAL: Reference-Grounded Reasoning for Multimodal Manipulation Detectioncited by 1
  32. 2026Impostor: An Agent-Curated Benchmark for Realistic AIGC Manipulation Localizationcited by 1
  33. 2026SPARED: Reasoning-Based AI-Generated Image Detection via Adversarially Edited Datacited by 1

Built on by

Not cited yet.