mlsanityData to 7 Oct 2026

Detecting and Grounding Multi-Modal Media Manipulation

arXiv 2304.02556 ↗First posted 5 Apr 2023cs.CVcited by 98

Not on the map. Why?

Abstract (arXiv)

Misinformation has become a pressing issue. Fake media, in both visual and textual forms, is widespread on the web. While various deepfake detection and text fake news detection methods have been proposed, they are only designed for single-modality forgery based on binary classification, let alone analyzing and reasoning subtle forgery traces across different modalities. In this paper, we highlight a new research problem for multi-modal fake media, namely Detecting and Grounding Multi-Modal Media Manipulation (DGM^4). DGM^4 aims to not only detect the authenticity of multi-modal media, but also ground the manipulated content (i.e., image bounding boxes and text tokens), which requires deeper reasoning of multi-modal media manipulation. To support a large-scale investigation, we construct the first DGM^4 dataset, where image-text pairs are manipulated by various approaches, with rich annotation of diverse manipulations. Moreover, we propose a novel HierArchical Multi-modal Manipulation rEasoning tRansformer (HAMMER) to fully capture the fine-grained interaction between different modalities. HAMMER performs 1) manipulation-aware contrastive learning between two uni-modal encoders as shallow manipulation reasoning, and 2) modality-aware cross-attention by multi-modal aggregator as deep manipulation reasoning. Dedicated manipulation detection and grounding heads are integrated from shallow to deep levels based on the interacted multi-modal information. Finally, we build an extensive benchmark and set up rigorous evaluation metrics for this new research problem. Comprehensive experiments demonstrate the superiority of our model; several valuable observations are also revealed to facilitate future research in multi-modal media manipulation.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Builds on 48 of its 65 references that are in the map. First posted on arXiv 5 Apr 2023.

Builds on

48 of 65 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  11. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
36 more
  1. 2019FaceForensics++: Learning to Detect Manipulated Facial Imagescited by 1,095
  2. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  3. 2019Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensicscited by 573
  4. 2019Defending Against Neural Fake Newscited by 553
  5. 2020Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Cluescited by 518
  6. 2019Face X-ray for More General Face Forgery Detectioncited by 469
  7. 2020Asymmetric Loss For Multi-Label Classificationcited by 379
  8. 2017"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detectioncited by 361
  9. 2021Multi-attentional Deepfake Detectioncited by 339
  10. 2017Fake News Detection on Social Media: A Data Mining Perspectivecited by 334
  11. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  12. 2021Generalizing Face Forgery Detection with High-frequency Featurescited by 261
  13. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  14. 2020DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detectioncited by 251
  15. 2021SimSwap: An Efficient Framework For High Fidelity Face Swappingcited by 243
  16. 2021Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domaincited by 241
  17. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  18. 2021High-Fidelity GAN Inversion for Image Attribute Editingcited by 197
  19. 2020Learning Self-Consistency for Deepfake Detectioncited by 179
  20. 2019Fourier Spectrum Discrepancies in Deep Network Generated Imagescited by 138
  21. 2021Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detectioncited by 133
  22. 2019Learning a Deep ConvNet for Multi-label Classification with Partial Labelscited by 124
  23. 2020Visual News: Benchmark and Challenges in News Image Captioningcited by 121
  24. 2021NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Mediacited by 106
  25. 2020NAS-FAS: Static-Dynamic Central Difference Network Search for Face Anti-Spoofingcited by 104
  26. 2021Local Relation Learning for Face Forgery Detectioncited by 100
  27. 2019Regularized Fine-grained Meta Face Anti-spoofingcited by 86
  28. 2019Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfercited by 81
  29. 2021Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resourcescited by 71
  30. 2021Exploiting Fine-grained Face Forgery Clues via Progressive Enhancement Learningcited by 60
  31. 2020Face Forgery Detection by 3D Decompositioncited by 53
  32. 2022Detecting and Recovering Sequential DeepFake Manipulationcited by 34
  33. 2020Open-set Adversarial Defensecited by 27
  34. 2022Open-set Adversarial Defense with Clean-Adversarial Mutual Learningcited by 21
  35. 2021Federated Generalized Face Presentation Attack Detectioncited by 9
  36. 2021Federated Test-Time Adaptive Face Presentation Attack Detection with Dual-Phase Privacy Preservationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 21; 2025: 36; 2026: 28.

98 papers.

Sort:

By area:

  1. 2023Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detectioncited by 180
  2. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 83
  3. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 80
  4. 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 73
  5. 2024SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detectioncited by 73
  6. 2025Spot the Fake: Large Multimodal Model-Based Synthetic Image Detection with Artifact Explanationcited by 66
  7. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  8. 2024MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMscited by 60
  9. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  10. 2023Detecting and Grounding Multi-Modal Media Manipulation and Beyondcited by 55
  11. 2023Multimodal Automated Fact-Checking: A Surveycited by 51
  12. 2023AutoSplice: A Text-prompt Manipulated Image Dataset for Media Forensicscited by 46
38 more
  1. 2024FIRE: Robust Detection of Diffusion-Generated Images via Frequency-Guided Reconstruction Errorcited by 46
  2. 2023Towards General Visual-Linguistic Face Forgery Detectioncited by 45
  3. 2024DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Expertscited by 42
  4. 2024FKA-Owl: Advancing Multimodal Fake News Detection through Knowledge-Augmented LVLMscited by 38
  5. 2025GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agentcited by 35
  6. 2024DiffusionFake: Enhancing Generalization in Deepfake Detection via Guided Stable Diffusioncited by 29
  7. 2023Frequency Domain-based Dataset Distillationcited by 28
  8. 2023DeepFake-Adapter: Dual-Level Adapter for DeepFake Detectioncited by 26
  9. 2025FALCON: Resolving Visual Redundancy and Fragmentation in High-resolution Multimodal Large Language Models via Visual Registerscited by 20
  10. 2025SemanticVLA: Semantic-Aligned Sparsification and Enhancement for Efficient Robotic Manipulationcited by 20
  11. 2025UniEmo: Unifying Emotional Understanding and Generation with Learnable Expert Queriescited by 18
  12. 2024Can LLMs Improve Multimodal Fact-Checking by Asking Relevant Questions?cited by 17
  13. 2025AVerImaTeC: A Dataset for Automatic Verification of Image-Text Claims with Evidence from the Webcited by 17
  14. 2026PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Recordscited by 16
  15. 2024MFC-Bench: Benchmarking Multimodal Fact-Checking with Large Vision-Language Modelscited by 15
  16. 2024MiRAGeNews: Multimodal Realistic AI-Generated News Detectioncited by 15
  17. 2023Unified Frequency-Assisted Transformer Framework for Detecting and Grounding Multi-Modal Manipulationcited by 14
  18. 2024Multimodal Misinformation Detection by Learning from Synthetic Data with Multimodal LLMscited by 13
  19. 2025Forensics-Bench: A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Modelscited by 13
  20. 2025HiconAgent: History Context-aware Policy Optimization for GUI Agentscited by 13
  21. 2025Survey on AI-Generated Media Detection: From Non-MLLM to MLLMcited by 12
  22. 2024Each Fake News is Fake in its Own Way: An Attribution Multi-Granularity Benchmark for Multimodal Fake News Detectioncited by 11
  23. 2025PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learningcited by 11
  24. 2025TRUST-VL: An Explainable News Assistant for General Multimodal Misinformation Detectioncited by 10
  25. 2026DeltaVLA: Prior-Guided Vision-Language-Action Models via World Knowledge Variationcited by 9
  26. 2026HATS: Hardness-Aware Trajectory Synthesis for GUI Agentscited by 9
  27. 2024Semantic Contextualization of Face Forgery: A New Definition, Dataset, and Detection Methodcited by 8
  28. 2025The Coherence Trap: When MLLM-Crafted Narratives Exploit Manipulated Visual Contextscited by 8
  29. 2023Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Groundingcited by 7
  30. 2024Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challengescited by 6
  31. 2026The Paradigm Shift: A Comprehensive Survey on Large Vision Language Models for Multimodal Fake News Detectioncited by 6
  32. 2026ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulationcited by 6
  33. 2024Fake News Detection and Manipulation Reasoning via Large Vision-Language Modelscited by 5
  34. 2025Multimodal Fake News Detection: MFND Dataset and Shallow-Deep Multitask Learningcited by 5
  35. 2025Seeing Through Deception: Uncovering Misleading Creator Intent in Multimodal News with Vision-Language Modelscited by 5
  36. 2025Unleashing the Potential of Consistency Learning for Detecting and Grounding Multi-Modal Media Manipulationcited by 5
  37. 2026VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checkingcited by 5
  38. 2026From Abstraction to Instantiation: Learning Behavioral Representation for Vision-Language-Action Modelcited by 5

50 most cited of 98 All 98 (JSON)