# GaLeNet: Multimodal Learning for Disaster Prediction, Management and Relief

arXiv: https://arxiv.org/abs/2206.09242 · HTML: https://mlsanity.com/paper/2206.09242/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 35 references that are in the map.
- First posted on arXiv 18 Jun 2022.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Revisiting Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/1901.09005/) (2019)
- [xBD: A Dataset for Assessing Building Damage from Satellite Imagery](https://mlsanity.com/paper/1911.09296/) (2019)
- [catch22: CAnonical Time-series CHaracteristics](https://mlsanity.com/paper/1901.10200/) (2019)
- [The Relative Performance of Ensemble Methods with Deep Convolutional Neural Networks for Image Classification](https://mlsanity.com/paper/1704.01664/) (2017)
- [Building Damage Detection in Satellite Imagery Using Convolutional Neural Networks](https://mlsanity.com/paper/1910.06444/) (2019)
- [Multimodal Categorization of Crisis Events in Social Media](https://mlsanity.com/paper/2004.04917/) (2020)
- [Building Disaster Damage Assessment in Satellite Imagery with Multi-Temporal Fusion](https://mlsanity.com/paper/2004.05525/) (2020)
- [RescueNet: Joint Building Segmentation and Damage Assessment from Satellite Imagery](https://mlsanity.com/paper/2004.07312/) (2020)
- [Cross-directional Feature Fusion Network for Building Damage Assessment from Satellite Imagery](https://mlsanity.com/paper/2010.14014/) (2020)

## Built on by (most cited first, 1 of 1)

- [AutoFraudNet: A Multimodal Network to Detect Fraud in the Auto Insurance Industry](https://mlsanity.com/paper/2301.07526/) (2023)

## Abstract (arXiv)

After a natural disaster, such as a hurricane, millions are left in need of emergency assistance. To allocate resources optimally, human planners need to accurately analyze data that can flow in large volumes from several sources. This motivates the development of multimodal machine learning frameworks that can integrate multiple data sources and leverage them efficiently. To date, the research community has mainly focused on unimodal reasoning to provide granular assessments of the damage. Moreover, previous studies mostly rely on post-disaster images, which may take several days to become available. In this work, we propose a multimodal framework (GaLeNet) for assessing the severity of damage by complementing pre-disaster images with weather data and the trajectory of the hurricane. Through extensive experiments on data from two hurricanes, we demonstrate (i) the merits of multimodal approaches compared to unimodal methods, and (ii) the effectiveness of GaLeNet at fusing various modalities. Furthermore, we show that GaLeNet can leverage pre-disaster images in the absence of post-disaster images, preventing substantial delays in decision making.