# Analysis of Social Media Data using Multimodal Deep Learning for Disaster Response

arXiv: https://arxiv.org/abs/2004.11838 · HTML: https://mlsanity.com/paper/2004.11838/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Apr 2020.

## Built on by (most cited first, 23 of 23)

- [HumAID: Human-Annotated Disaster Incidents Data from Twitter with Deep Learning Benchmarks](https://mlsanity.com/paper/2104.03090/) (2021)
- [Cross-Modal Projection in Multimodal LLMs Doesn't Really Project Visual Attributes to Textual Space](https://mlsanity.com/paper/2402.16832/) (2024)
- [MEDIC: A Multi-Task Learning Dataset for Disaster Image Classification](https://mlsanity.com/paper/2108.12828/) (2021)
- [Overcoming Language Disparity in Online Content Classification with Multimodal Learning](https://mlsanity.com/paper/2205.09744/) (2022)
- [CrowdAgent: Multi-Agent Managed Multi-Source Annotation System](https://mlsanity.com/paper/2509.14030/) (2025)
- [Robust Training of Social Media Image Classification Models for Rapid Disaster Response](https://mlsanity.com/paper/2104.04184/) (2021)
- [Robustness of Fusion-based Multimodal Classifiers to Cross-Modal Content Dilutions](https://mlsanity.com/paper/2211.02646/) (2022)
- [Few-shot Learning for Multi-modal Social Media Event Filtering](https://mlsanity.com/paper/2211.10340/) (2022)
- [Zero-Shot Classification of Crisis Tweets Using Instruction-Finetuned Large Language Models](https://mlsanity.com/paper/2410.00182/) (2024)
- [Landslide Detection in Real-Time Social Media Image Streams](https://mlsanity.com/paper/2110.04080/) (2021)
- [A Real-time System for Detecting Landslide Reports on Social Media using Artificial Intelligence](https://mlsanity.com/paper/2202.07475/) (2022)
- [TeamX@DravidianLangTech-ACL2022: A Comparative Analysis for Troll-Based Meme Classification](https://mlsanity.com/paper/2205.04404/) (2022)
- [More is Less? A Simulation-Based Approach to Dynamic Interactions between Biases in Multimodal Models](https://mlsanity.com/paper/2412.17505/) (2024)
- [MMiC: Mitigating Modality Incompleteness in Clustered Federated Learning](https://mlsanity.com/paper/2505.06911/) (2025)
- [MultiMatch: Multihead Consistency Regularization Matching for Semi-Supervised Text Classification](https://mlsanity.com/paper/2506.07801/) (2025)
- [A Machine learning approach for rapid disaster response based on multi-modal data. The case of housing & shelter needs](https://mlsanity.com/paper/2108.00887/) (2021)
- [WildFireCan-MMD: A Multimodal Dataset for Classification of User-Generated Content During Wildfires in Canada](https://mlsanity.com/paper/2504.13231/) (2025)
- [Contextual Attention-Based Multimodal Fusion of LLM and CNN for Sentiment Analysis](https://mlsanity.com/paper/2508.13196/) (2025)
- [Structured AI Decision-Making in Disaster Management](https://mlsanity.com/paper/2509.01576/) (2025)
- [Multimodal Learning with Augmentation Techniques for Natural Disaster Assessment](https://mlsanity.com/paper/2511.00004/) (2025)
- [Cross-Modal Rationale Transfer for Explainable Humanitarian Classification on Social Media](https://mlsanity.com/paper/2603.18611/) (2026)
- [Theia: Large-Scale Multimodal Captioning and Automated Validation of the Incidents1M Dataset for Data-Free Distillation](https://mlsanity.com/paper/2607.28269/) (2026)
- [Codebook-Guided Cross-Modal Knowledge Distillation for Structurally Heterogeneous Features](https://mlsanity.com/paper/2609.37243/) (2026)

## Abstract (arXiv)

Multimedia content in social media platforms provides significant information during disaster events. The types of information shared include reports of injured or deceased people, infrastructure damage, and missing or found people, among others. Although many studies have shown the usefulness of both text and image content for disaster response purposes, the research has been mostly focused on analyzing only the text modality in the past. In this paper, we propose to use both text and image modalities of social media data to learn a joint representation using state-of-the-art deep learning techniques. Specifically, we utilize convolutional neural networks to define a multimodal deep learning architecture with a modality-agnostic shared representation. Extensive experiments on real-world disaster datasets show that the proposed multimodal architecture yields better performance than models trained using a single modality (e.g., either text or image).