# CrisisKAN: Knowledge-infused and Explainable Multimodal Attention Network for Crisis Event Classification

arXiv: https://arxiv.org/abs/2401.06194 · HTML: https://mlsanity.com/paper/2401.06194/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 44 references that are in the map.
- First posted on arXiv 11 Jan 2024.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training](https://mlsanity.com/paper/1908.06066/) (2019)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [GhostNet: More Features from Cheap Operations](https://mlsanity.com/paper/1911.11907/) (2019)
- [Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers](https://mlsanity.com/paper/2004.00849/) (2020)
- [One weird trick for parallelizing convolutional neural networks](https://mlsanity.com/paper/1404.5997/) (2014)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [Towards Robust Vision Transformer](https://mlsanity.com/paper/2105.07926/) (2021)
- [Supervised Multimodal Bitransformers for Classifying Images and Text](https://mlsanity.com/paper/1909.02950/) (2019)
- [CrisisMMD: Multimodal Twitter Datasets from Natural Disasters](https://mlsanity.com/paper/1805.00713/) (2018)
- [CentralNet: a Multilayer Approach for Multimodal Fusion](https://mlsanity.com/paper/1808.07275/) (2018)
- [Black-box Explanation of Object Detectors via Saliency Maps](https://mlsanity.com/paper/2006.03204/) (2020)
- [Efficient Large-Scale Multi-Modal Classification](https://mlsanity.com/paper/1802.02892/) (2018)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [The Role of User Profile for Fake News Detection](https://mlsanity.com/paper/1904.13355/) (2019)
- [Multimodal Categorization of Crisis Events in Social Media](https://mlsanity.com/paper/2004.04917/) (2020)
- [Understanding Public Sentiments, Opinions and Topics about COVID-19 using Twitter](https://mlsanity.com/paper/2012.03039/) (2020)
- [Image-Text Pre-Training for Logo Recognition](https://mlsanity.com/paper/2309.10206/) (2023)

## Built on by (most cited first, 6 of 6)

- [MMiC: Mitigating Modality Incompleteness in Clustered Federated Learning](https://mlsanity.com/paper/2505.06911/) (2025)
- [WildFireCan-MMD: A Multimodal Dataset for Classification of User-Generated Content During Wildfires in Canada](https://mlsanity.com/paper/2504.13231/) (2025)
- [Differential Attention for Multimodal Crisis Event Analysis](https://mlsanity.com/paper/2507.05165/) (2025)
- [Multimodal Learning with Augmentation Techniques for Natural Disaster Assessment](https://mlsanity.com/paper/2511.00004/) (2025)
- [Cross-Modal Rationale Transfer for Explainable Humanitarian Classification on Social Media](https://mlsanity.com/paper/2603.18611/) (2026)
- [Self-Supervised Visual Representation Learning: Pretrain-Finetuning or Joint Training?](https://mlsanity.com/paper/2607.13192/) (2026)

## Abstract (arXiv)

Pervasive use of social media has become the emerging source for real-time information (like images, text, or both) to identify various events. Despite the rapid growth of image and text-based event classification, the state-of-the-art (SOTA) models find it challenging to bridge the semantic gap between features of image and text modalities due to inconsistent encoding. Also, the black-box nature of models fails to explain the model's outcomes for building trust in high-stakes situations such as disasters, pandemic. Additionally, the word limit imposed on social media posts can potentially introduce bias towards specific events. To address these issues, we proposed CrisisKAN, a novel Knowledge-infused and Explainable Multimodal Attention Network that entails images and texts in conjunction with external knowledge from Wikipedia to classify crisis events. To enrich the context-specific understanding of textual information, we integrated Wikipedia knowledge using proposed wiki extraction algorithm. Along with this, a guided cross-attention module is implemented to fill the semantic gap in integrating visual and textual data. In order to ensure reliability, we employ a model-specific approach called Gradient-weighted Class Activation Mapping (Grad-CAM) that provides a robust explanation of the predictions of the proposed model. The comprehensive experiments conducted on the CrisisMMD dataset yield in-depth analysis across various crisis-specific tasks and settings. As a result, CrisisKAN outperforms existing SOTA methodologies and provides a novel view in the domain of explainable multimodal event classification.