# Investigating Vulnerability to Adversarial Examples on Multimodal Data Fusion in Deep Learning

arXiv: https://arxiv.org/abs/2005.10987 · HTML: https://mlsanity.com/paper/2005.10987/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2020.

## Built on by (most cited first, 11 of 11)

- [Exploring Adversarial Robustness of Multi-Sensor Perception Systems in Self Driving](https://mlsanity.com/paper/2101.06784/) (2021)
- [Contrastive language and vision learning of general fashion concepts](https://mlsanity.com/paper/2204.03972/) (2022)
- [Dual-Key Multimodal Backdoors for Visual Question Answering](https://mlsanity.com/paper/2112.07668/) (2021)
- [Quantifying and Enhancing Multi-modal Robustness with Modality Preference](https://mlsanity.com/paper/2402.06244/) (2024)
- [What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?](https://mlsanity.com/paper/2310.06383/) (2023)
- [Adversarial Vulnerability of Temporal Feature Networks for Object Detection](https://mlsanity.com/paper/2208.10773/) (2022)
- [AI Safety in Practice: Enhancing Adversarial Robustness in Multimodal Image Captioning](https://mlsanity.com/paper/2407.21174/) (2024)
- [Hardening RGB-D Object Recognition Systems against Adversarial Patch Attacks](https://mlsanity.com/paper/2309.07106/) (2023)
- [Quantifying the robustness of deep multispectral segmentation models against natural perturbations and data poisoning](https://mlsanity.com/paper/2305.11347/) (2023)
- [Impact of architecture on robustness and interpretability of multispectral deep neural networks](https://mlsanity.com/paper/2309.12463/) (2023)
- [Revisiting Misalignment in Multispectral Pedestrian Detection: A Language-Driven Approach for Cross-modal Alignment Fusion](https://mlsanity.com/paper/2411.17995/) (2024)

## Abstract (arXiv)

The success of multimodal data fusion in deep learning appears to be attributed to the use of complementary in-formation between multiple input data. Compared to their predictive performance, relatively less attention has been devoted to the robustness of multimodal fusion models. In this paper, we investigated whether the current multimodal fusion model utilizes the complementary intelligence to defend against adversarial attacks. We applied gradient based white-box attacks such as FGSM and PGD on MFNet, which is a major multispectral (RGB, Thermal) fusion deep learning model for semantic segmentation. We verified that the multimodal fusion model optimized for better prediction is still vulnerable to adversarial attack, even if only one of the sensors is attacked. Thus, it is hard to say that existing multimodal data fusion models are fully utilizing complementary relationships between multiple modalities in terms of adversarial robustness. We believe that our observations open a new horizon for adversarial attack research on multimodal data fusion.