# Detecting Medical Misinformation on Social Media Using Multimodal Deep Learning

arXiv: https://arxiv.org/abs/2012.13968 · HTML: https://mlsanity.com/paper/2012.13968/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Dec 2020.

## Built on by (most cited first, 9 of 9)

- [MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets](https://mlsanity.com/paper/2109.05184/) (2021)
- [Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2303.13948/) (2023)
- [Multi-modal Misinformation Detection: Approaches, Challenges and Opportunities](https://mlsanity.com/paper/2203.13883/) (2022)
- [Detecting Misinformation in Multimedia Content through Cross-Modal Entity Consistency: A Dual Learning Approach](https://mlsanity.com/paper/2409.00022/) (2024)
- [Combating Health Misinformation in Social Media: Characterization, Detection, Intervention, and Open Issues](https://mlsanity.com/paper/2211.05289/) (2022)
- [Detecting Anti-Vaccine Users on Twitter](https://mlsanity.com/paper/2110.11333/) (2021)
- [A Scoping Review of Natural Language Processing in Addressing Medically Inaccurate Information: Errors, Misinformation, and Hallucination](https://mlsanity.com/paper/2505.00008/) (2025)
- [Insta-VAX: A Multimodal Benchmark for Anti-Vaccine and Misinformation Posts Detection on Social Media](https://mlsanity.com/paper/2112.08470/) (2021)
- [Forecasting User Interests Through Topic Tag Predictions in Online Health Communities](https://mlsanity.com/paper/2211.02789/) (2022)

## Abstract (arXiv)

In 2019, outbreaks of vaccine-preventable diseases reached the highest number in the US since 1992. Medical misinformation, such as antivaccine content propagating through social media, is associated with increases in vaccine delay and refusal. Our overall goal is to develop an automatic detector for antivaccine messages to counteract the negative impact that antivaccine messages have on the public health. Very few extant detection systems have considered multimodality of social media posts (images, texts, and hashtags), and instead focus on textual components, despite the rapid growth of photo-sharing applications (e.g., Instagram). As a result, existing systems are not sufficient for detecting antivaccine messages with heavy visual components (e.g., images) posted on these newer platforms. To solve this problem, we propose a deep learning network that leverages both visual and textual information. A new semantic- and task-level attention mechanism was created to help our model to focus on the essential contents of a post that signal antivaccine messages. The proposed model, which consists of three branches, can generate comprehensive fused features for predictions. Moreover, an ensemble method is proposed to further improve the final prediction accuracy. To evaluate the proposed model's performance, a real-world social media dataset that consists of more than 30,000 samples was collected from Instagram between January 2016 and October 2019. Our 30 experiment results demonstrate that the final network achieves above 97% testing accuracy and outperforms other relevant models, demonstrating that it can detect a large amount of antivaccine messages posted daily. The implementation code is available at https://github.com/wzhings/antivaccine_detection.