# Impact of architecture on robustness and interpretability of multispectral deep neural networks

arXiv: https://arxiv.org/abs/2309.12463 · HTML: https://mlsanity.com/paper/2309.12463/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming](https://mlsanity.com/paper/1907.07484/) (2019)
- [Counterfactual Visual Explanations](https://mlsanity.com/paper/1904.07451/) (2019)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization](https://mlsanity.com/paper/2107.04649/) (2021)
- [SpaceNet: A Remote Sensing Dataset and Challenge Series](https://mlsanity.com/paper/1807.01232/) (2018)
- [A Universal Law of Robustness via Isoperimetry](https://mlsanity.com/paper/2105.12806/) (2021)
- [Explaining Image Classifiers by Counterfactual Generation](https://mlsanity.com/paper/1807.08024/) (2018)
- [Benchmarking the Robustness of Semantic Segmentation Models](https://mlsanity.com/paper/1908.05005/) (2019)
- [Motivating the Rules of the Game for Adversarial Example Research](https://mlsanity.com/paper/1807.06732/) (2018)
- [Semantic Stereo for Incidental Satellite Images](https://mlsanity.com/paper/1811.08739/) (2018)
- [RarePlanes: Synthetic Data Takes Flight](https://mlsanity.com/paper/2006.02963/) (2020)
- [Missingness Bias in Model Debugging](https://mlsanity.com/paper/2204.08945/) (2022)
- [Perceptual Score: What Data Modalities Does Your Model Perceive?](https://mlsanity.com/paper/2110.14375/) (2021)
- [Learning Geocentric Object Pose in Oblique Monocular Images](https://mlsanity.com/paper/2007.00729/) (2020)
- [Investigating Vulnerability to Adversarial Examples on Multimodal Data Fusion in Deep Learning](https://mlsanity.com/paper/2005.10987/) (2020)
- [Adversarial Attacks against a Satellite-borne Multispectral Cloud Detector](https://mlsanity.com/paper/2112.01723/) (2021)
- [Single View Geocentric Pose in the Wild](https://mlsanity.com/paper/2105.08229/) (2021)
- [Quantifying the robustness of deep multispectral segmentation models against natural perturbations and data poisoning](https://mlsanity.com/paper/2305.11347/) (2023)

## Abstract (arXiv)

Including information from additional spectral bands (e.g., near-infrared) can improve deep learning model performance for many vision-oriented tasks. There are many possible ways to incorporate this additional information into a deep learning model, but the optimal fusion strategy has not yet been determined and can vary between applications. At one extreme, known as "early fusion," additional bands are stacked as extra channels to obtain an input image with more than three channels. At the other extreme, known as "late fusion," RGB and non-RGB bands are passed through separate branches of a deep learning model and merged immediately before a final classification or segmentation layer. In this work, we characterize the performance of a suite of multispectral deep learning models with different fusion approaches, quantify their relative reliance on different input bands and evaluate their robustness to naturalistic image corruptions affecting one or more input channels.