# Investigating Imbalances Between SAR and Optical Utilization for Multi-Modal Urban Mapping

arXiv: https://arxiv.org/abs/2304.05080 · HTML: https://mlsanity.com/paper/2304.05080/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 14 references that are in the map.
- First posted on arXiv 11 Apr 2023.

## Builds on (most cited first, 6 of 6)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [MMTM: Multimodal Transfer Module for CNN Fusion](https://mlsanity.com/paper/1911.08670/) (2019)
- [Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks](https://mlsanity.com/paper/2202.05306/) (2022)
- [Weakly Supervised Semantic Segmentation of Satellite Images for Land Cover Mapping -- Challenges and Opportunities](https://mlsanity.com/paper/2002.08254/) (2020)
- [Convolutional Neural Networks for Global Human Settlements Mapping from Sentinel-2 Satellite Imagery](https://mlsanity.com/paper/2006.03267/) (2020)
- [A framework for large-scale mapping of human settlement extent from Sentinel-2 images via fully convolutional neural networks](https://mlsanity.com/paper/2001.11935/) (2020)

## Abstract (arXiv)

Accurate urban maps provide essential information to support sustainable urban development. Recent urban mapping methods use multi-modal deep neural networks to fuse Synthetic Aperture Radar (SAR) and optical data. However, multi-modal networks may rely on just one modality due to the greedy nature of learning. In turn, the imbalanced utilization of modalities can negatively affect the generalization ability of a network. In this paper, we investigate the utilization of SAR and optical data for urban mapping. To that end, a dual-branch network architecture using intermediate fusion modules to share information between the uni-modal branches is utilized. A cut-off mechanism in the fusion modules enables the stopping of information flow between the branches, which is used to estimate the network's dependence on SAR and optical data. While our experiments on the SEN12 Global Urban Mapping dataset show that good performance can be achieved with conventional SAR-optical data fusion (F1 score = 0.682 ± 0.014), we also observed a clear under-utilization of optical data. Therefore, future work is required to investigate whether a more balanced utilization of SAR and optical data can lead to performance improvements.