# A Survey on Multimodal Disinformation Detection

arXiv: https://arxiv.org/abs/2103.12541 · HTML: https://mlsanity.com/paper/2103.12541/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 168 references that are in the map.
- First posted on arXiv 13 Mar 2021.

## Builds on (most cited first, 30 of 67)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://mlsanity.com/paper/1909.02164/) (2019)
- [Fake News Detection on Social Media: A Data Mining Perspective](https://mlsanity.com/paper/1708.01967/) (2017)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [Fine-Grained Analysis of Propaganda in News Articles](https://mlsanity.com/paper/1910.02517/) (2019)
- [Generating Fact Checking Explanations](https://mlsanity.com/paper/2004.05773/) (2020)
- [Supervised Multimodal Bitransformers for Classifying Images and Text](https://mlsanity.com/paper/1909.02950/) (2019)
- [Understanding Abuse: A Typology of Abusive Language Detection Subtasks](https://mlsanity.com/paper/1705.09899/) (2017)
- [Exploring Hate Speech Detection in Multimodal Publications](https://mlsanity.com/paper/1910.03814/) (2019)
- [FANG: Leveraging Social Context for Fake News Detection Using Graph Representation](https://mlsanity.com/paper/2008.07939/) (2020)
- [Automated Fact Checking: Task formulations, methods and future directions](https://mlsanity.com/paper/1806.07687/) (2018)
- [Beyond News Contents: The Role of Social Context for Fake News Detection](https://mlsanity.com/paper/1712.07709/) (2017)
- [Fighting an Infodemic: COVID-19 Fake News Dataset](https://mlsanity.com/paper/2011.03327/) (2020)
- [Explainable Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2011.03870/) (2020)
- [That is a Known Lie: Detecting Previously Fact-Checked Claims](https://mlsanity.com/paper/2005.06058/) (2020)
- [The spread of low-credibility content by social bots](https://mlsanity.com/paper/1707.07592/) (2017)
- [Fighting the COVID-19 Infodemic: Modeling the Perspective of Journalists, Fact-Checkers, Social Media Platforms, Policy Makers, and the Society](https://mlsanity.com/paper/2005.00033/) (2020)
- [Predicting Factuality of Reporting and Bias of News Media Sources](https://mlsanity.com/paper/1810.01765/) (2018)
- [A Survey on Computational Propaganda Detection](https://mlsanity.com/paper/2007.08024/) (2020)
- [Deep Audio-Visual Learning: A Survey](https://mlsanity.com/paper/2001.04758/) (2020)
- [UKP-Athene: Multi-Sentence Textual Entailment for Claim Verification](https://mlsanity.com/paper/1809.01479/) (2018)
- [Spread of hate speech in online social media](https://mlsanity.com/paper/1812.01693/) (2018)
- [Characterizing and Detecting Hateful Users on Twitter](https://mlsanity.com/paper/1803.08977/) (2018)
- [r/Fakeddit: A New Multimodal Benchmark Dataset for Fine-grained Fake News Detection](https://mlsanity.com/paper/1911.03854/) (2019)
- [CPTR: Full Transformer Network for Image Captioning](https://mlsanity.com/paper/2101.10804/) (2021)
- [Fact-Checking Meets Fauxtography: Verifying Claims About Images](https://mlsanity.com/paper/1908.11722/) (2019)

## Built on by (most cited first, 30 of 47)

- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Detecting Harmful Memes and Their Targets](https://mlsanity.com/paper/2110.00413/) (2021)
- [Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems](https://mlsanity.com/paper/2401.05778/) (2024)
- [Detecting and Understanding Harmful Memes: A Survey](https://mlsanity.com/paper/2205.04274/) (2022)
- [Multimodal Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2305.13507/) (2023)
- [VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Bias](https://mlsanity.com/paper/2304.14133/) (2023)
- [Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking](https://mlsanity.com/paper/2301.11843/) (2023)
- [Self-Checker: Plug-and-Play Modules for Fact-Checking with Large Language Models](https://mlsanity.com/paper/2305.14623/) (2023)
- [Synthetic Misinformers: Generating and Combating Multimodal Misinformation](https://mlsanity.com/paper/2303.01217/) (2023)
- [Can GPT-4 Identify Propaganda? Annotation and Detection of Propaganda Spans in News Articles](https://mlsanity.com/paper/2402.17478/) (2024)
- [DISARM: Detecting the Victims Targeted by Harmful Memes](https://mlsanity.com/paper/2205.05738/) (2022)
- [Assisting the Human Fact-Checkers: Detecting All Previously Fact-Checked Claims in a Document](https://mlsanity.com/paper/2109.07410/) (2021)
- [Overview of the WANLP 2022 Shared Task on Propaganda Detection in Arabic](https://mlsanity.com/paper/2211.10057/) (2022)
- [RED-DOT: Multimodal Fact-checking via Relevant Evidence Detection](https://mlsanity.com/paper/2311.09939/) (2023)
- [Interpretable Multimodal Misinformation Detection with Logic Reasoning](https://mlsanity.com/paper/2305.05964/) (2023)
- [Multi-modal Misinformation Detection: Approaches, Challenges and Opportunities](https://mlsanity.com/paper/2203.13883/) (2022)
- [Combating Online Misinformation Videos: Characterization, Detection, and Future Directions](https://mlsanity.com/paper/2302.03242/) (2023)
- [DisinfoMeme: A Multimodal Dataset for Detecting Meme Intentionally Spreading Out Disinformation](https://mlsanity.com/paper/2205.12617/) (2022)
- [Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicities](https://mlsanity.com/paper/2406.07353/) (2024)
- [Dataset of Fake News Detection and Fact Verification: A Survey](https://mlsanity.com/paper/2111.03299/) (2021)
- [The DSA Transparency Database: Auditing Self-reported Moderation Actions by Social Media](https://mlsanity.com/paper/2312.10269/) (2023)
- [CrowdChecked: Detecting Previously Fact-Checked Claims in Social Media](https://mlsanity.com/paper/2210.04447/) (2022)
- [Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI](https://mlsanity.com/paper/2308.07213/) (2023)
- [Native vs Non-Native Language Prompting: A Comparative Analysis](https://mlsanity.com/paper/2409.07054/) (2024)
- [Credible, Unreliable or Leaked?: Evidence Verification for Enhanced Automated Fact-checking](https://mlsanity.com/paper/2404.18971/) (2024)
- [Propaganda to Hate: A Multimodal Analysis of Arabic Memes with Multi-Agent LLMs](https://mlsanity.com/paper/2409.07246/) (2024)
- [Findings of Factify 2: Multimodal Fake News Detection](https://mlsanity.com/paper/2307.10475/) (2023)
- [Datavoidant: An AI System for Addressing Political Data Voids on Social Media](https://mlsanity.com/paper/2210.13594/) (2022)
- [GREENER: Graph Neural Networks for News Media Profiling](https://mlsanity.com/paper/2211.05533/) (2022)
- [Seeing Through Deception: Uncovering Misleading Creator Intent in Multimodal News with Vision-Language Models](https://mlsanity.com/paper/2505.15489/) (2025)

## Abstract (arXiv)

Recent years have witnessed the proliferation of offensive content online such as fake news, propaganda, misinformation, and disinformation. While initially this was mostly about textual content, over time images and videos gained popularity, as they are much easier to consume, attract more attention, and spread further than text. As a result, researchers started leveraging different modalities and combinations thereof to tackle online multimodal offensive content. In this study, we offer a survey on the state-of-the-art on multimodal disinformation detection covering various combinations of modalities: text, images, speech, video, social media network structure, and temporal information. Moreover, while some studies focused on factuality, others investigated how harmful the content is. While these two components in the definition of disinformation (i) factuality, and (ii) harmfulness, are equally important, they are typically studied in isolation. Thus, we argue for the need to tackle disinformation detection by taking into account multiple modalities as well as both factuality and harmfulness, in the same framework. Finally, we discuss current challenges and future research directions