# Multimodal Categorization of Crisis Events in Social Media

arXiv: https://arxiv.org/abs/2004.04917 · HTML: https://mlsanity.com/paper/2004.04917/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2020.

## Built on by (most cited first, 30 of 34)

- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [Multimodal Knowledge Expansion](https://mlsanity.com/paper/2103.14431/) (2021)
- [Incidents1M: a large-scale dataset of images with natural disasters, damage, and incidents](https://mlsanity.com/paper/2201.04236/) (2022)
- [Harnessing Large Language Models for Disaster Management: A Survey](https://mlsanity.com/paper/2501.06932/) (2025)
- [Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoning](https://mlsanity.com/paper/2503.12972/) (2025)
- [Multimodal Fake News Detection](https://mlsanity.com/paper/2112.04831/) (2021)
- [Adaptive Contrastive Learning on Multimodal Transformer for Review Helpfulness Predictions](https://mlsanity.com/paper/2211.03524/) (2022)
- [SANCL: Multimodal Review Helpfulness Prediction with Selective Attention and Natural Contrastive Learning](https://mlsanity.com/paper/2209.05040/) (2022)
- [A social context-aware graph-based multimodal attentive learning framework for disaster content classification during emergencies: a benchmark dataset and method](https://mlsanity.com/paper/2410.08814/) (2024)
- [Gradient-Boosted Decision Tree for Listwise Context Model in Multimodal Review Helpfulness Prediction](https://mlsanity.com/paper/2305.12678/) (2023)
- [Understanding Social Media Cross-Modality Discourse in Linguistic Space](https://mlsanity.com/paper/2302.13311/) (2023)
- [Overcome Modal Bias in Multi-modal Federated Learning via Balanced Modality Selection](https://mlsanity.com/paper/2401.00403/) (2023)
- [CrisisKAN: Knowledge-infused and Explainable Multimodal Attention Network for Crisis Event Classification](https://mlsanity.com/paper/2401.06194/) (2024)
- [Robust Training of Social Media Image Classification Models for Rapid Disaster Response](https://mlsanity.com/paper/2104.04184/) (2021)
- [Resolving Sentiment Discrepancy for Multimodal Sentiment Detection via Semantics Completion and Decomposition](https://mlsanity.com/paper/2407.07026/) (2024)
- [VIDI: A Video Dataset of Incidents](https://mlsanity.com/paper/2205.13277/) (2022)
- [Harnessing Shared Relations via Multimodal Mixup Contrastive Learning for Multimodal Classification](https://mlsanity.com/paper/2409.17777/) (2024)
- [Multimodal Breast Lesion Classification Using Cross-Attention Deep Networks](https://mlsanity.com/paper/2108.09591/) (2021)
- [GaLeNet: Multimodal Learning for Disaster Prediction, Management and Relief](https://mlsanity.com/paper/2206.09242/) (2022)
- [Balanced Multi-modal Federated Learning via Cross-Modal Infiltration](https://mlsanity.com/paper/2401.00894/) (2023)
- [More is Less? A Simulation-Based Approach to Dynamic Interactions between Biases in Multimodal Models](https://mlsanity.com/paper/2412.17505/) (2024)
- [Efficient Traffic Forecasting on Large-Scale Road Network by Regularized Adaptive Graph Convolution](https://mlsanity.com/paper/2506.07179/) (2025)
- [CAMO: Causality-Guided Adversarial Multimodal Domain Generalization for Crisis Classification](https://mlsanity.com/paper/2512.08071/) (2025)
- [Interactive Event Sifting using Bayesian Graph Neural Networks](https://mlsanity.com/paper/2410.05359/) (2024)
- [WildFireCan-MMD: A Multimodal Dataset for Classification of User-Generated Content During Wildfires in Canada](https://mlsanity.com/paper/2504.13231/) (2025)
- [PREMISE: Matching-based Prediction for Accurate Review Recommendation](https://mlsanity.com/paper/2505.01255/) (2025)
- [Differential Attention for Multimodal Crisis Event Analysis](https://mlsanity.com/paper/2507.05165/) (2025)
- [Multimodal Fine-grained Reasoning for Post Quality Evaluation](https://mlsanity.com/paper/2507.17934/) (2025)
- [Contextual Attention-Based Multimodal Fusion of LLM and CNN for Sentiment Analysis](https://mlsanity.com/paper/2508.13196/) (2025)
- [Cross-Attention Multimodal Fusion for Breast Cancer Diagnosis: Integrating Mammography and Clinical Data with Explainability](https://mlsanity.com/paper/2508.16000/) (2025)

## Abstract (arXiv)

Recent developments in image classification and natural language processing, coupled with the rapid growth in social media usage, have enabled fundamental advances in detecting breaking events around the world in real-time. Emergency response is one such area that stands to gain from these advances. By processing billions of texts and images a minute, events can be automatically detected to enable emergency response workers to better assess rapidly evolving situations and deploy resources accordingly. To date, most event detection techniques in this area have focused on image-only or text-only approaches, limiting detection performance and impacting the quality of information delivered to crisis response teams. In this paper, we present a new multimodal fusion method that leverages both images and texts as input. In particular, we introduce a cross-attention module that can filter uninformative and misleading components from weak modalities on a sample by sample basis. In addition, we employ a multimodal graph-based approach to stochastically transition between embeddings of different multimodal pairs during training to better regularize the learning process as well as dealing with limited training data by constructing new matched pairs from different samples. We show that our method outperforms the unimodal approaches and strong multimodal baselines by a large margin on three crisis-related tasks.