# Beyond RGB: Very High Resolution Urban Remote Sensing With Multimodal Deep Networks

arXiv: https://arxiv.org/abs/1711.08681 · HTML: https://mlsanity.com/paper/1711.08681/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2017.

## Built on by (most cited first, 30 of 64)

- [UNetFormer: A UNet-like Transformer for Efficient Semantic Segmentation of Remote Sensing Urban Scene Imagery](https://mlsanity.com/paper/2109.08937/) (2021)
- [TPH-YOLOv5: Improved YOLOv5 Based on Transformer Prediction Head for Object Detection on Drone-captured Scenarios](https://mlsanity.com/paper/2108.11539/) (2021)
- [A Novel Transformer Based Semantic Segmentation Scheme for Fine-Resolution Remote Sensing Images](https://mlsanity.com/paper/2104.12137/) (2021)
- [Cross-City Matters: A Multimodal Remote Sensing Benchmark Dataset for Cross-City Semantic Segmentation using High-Resolution Domain Adaptation Networks](https://mlsanity.com/paper/2309.16499/) (2023)
- [SAM-Assisted Remote Sensing Imagery Semantic Segmentation with Object and Boundary Constraints](https://mlsanity.com/paper/2312.02464/) (2023)
- [Transformer Meets Convolution: A Bilateral Awareness Network for Semantic Segmentation of Very Fine Resolution Urban Scene Images](https://mlsanity.com/paper/2106.12413/) (2021)
- [GLF-CR: SAR-Enhanced Cloud Removal with Global-Local Fusion](https://mlsanity.com/paper/2206.02850/) (2022)
- [PopulAtion Parameter Averaging (PAPA)](https://mlsanity.com/paper/2304.03094/) (2023)
- [FedDiff: Diffusion Model Driven Federated Learning for Multi-Modal and Multi-Clients](https://mlsanity.com/paper/2401.02433/) (2023)
- [ABCNet: Attentive Bilateral Contextual Network for Efficient Semantic Segmentation of Fine-Resolution Remote Sensing Images](https://mlsanity.com/paper/2102.02531/) (2021)
- [Current Trends in Deep Learning for Earth Observation: An Open-source Benchmark Arena for Image Classification](https://mlsanity.com/paper/2207.07189/) (2022)
- [Multi-modal learning for geospatial vegetation forecasting](https://mlsanity.com/paper/2303.16198/) (2023)
- [RingMoE: Mixture-of-Modality-Experts Multi-Modal Foundation Models for Universal Remote Sensing Image Interpretation](https://mlsanity.com/paper/2504.03166/) (2025)
- [A Unified Framework with Multimodal Fine-tuning for Remote Sensing Semantic Segmentation](https://mlsanity.com/paper/2410.11160/) (2024)
- [HED-UNet: Combined Segmentation and Edge Detection for Monitoring the Antarctic Coastline](https://mlsanity.com/paper/2103.01849/) (2021)
- [Curriculum-style Local-to-global Adaptation for Cross-domain Remote Sensing Image Segmentation](https://mlsanity.com/paper/2203.01539/) (2022)
- [Large-scale Building Height Retrieval from Single SAR Imagery based on Bounding Box Regression Networks](https://mlsanity.com/paper/2111.09460/) (2021)
- [Weakly Supervised Change Detection Using Guided Anisotropic Difusion](https://mlsanity.com/paper/2112.15367/) (2021)
- [Augmentation Invariance and Adaptive Sampling in Semantic Segmentation of Agricultural Aerial Images](https://mlsanity.com/paper/2204.07969/) (2022)
- [FedFusion: Manifold Driven Federated Learning for Multi-satellite and Multi-modality Fusion](https://mlsanity.com/paper/2311.09540/) (2023)
- [Adaptive Fusion of Multi-view Remote Sensing data for Optimal Sub-field Crop Yield Prediction](https://mlsanity.com/paper/2401.11844/) (2024)
- [MultiScene: A Large-scale Dataset and Benchmark for Multi-scene Recognition in Single Aerial Images](https://mlsanity.com/paper/2104.02846/) (2021)
- [An Attention-Fused Network for Semantic Segmentation of Very-High-Resolution Remote Sensing Imagery](https://mlsanity.com/paper/2105.04132/) (2021)
- [No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths](https://mlsanity.com/paper/2306.11922/) (2023)
- [A Contrastive Distillation Approach for Incremental Semantic Segmentation in Aerial Images](https://mlsanity.com/paper/2112.03814/) (2021)
- [MetaSegNet: Metadata-collaborative Vision-Language Representation Learning for Semantic Segmentation of Remote Sensing Images](https://mlsanity.com/paper/2312.12735/) (2023)
- [Scale-aware Neural Network for Semantic Segmentation of Multi-resolution Remote Sensing Images](https://mlsanity.com/paper/2103.07935/) (2021)
- [Single-stream CNN with Learnable Architecture for Multi-source Remote Sensing Data](https://mlsanity.com/paper/2109.06094/) (2021)
- [Hierarchical Instance Mixing across Domains in Aerial Segmentation](https://mlsanity.com/paper/2210.06216/) (2022)
- [Common Practices and Taxonomy in Deep Multi-view Fusion for Remote Sensing Applications](https://mlsanity.com/paper/2301.01200/) (2022)

## Abstract (arXiv)

In this work, we investigate various methods to deal with semantic labeling of very high resolution multi-modal remote sensing data. Especially, we study how deep fully convolutional networks can be adapted to deal with multi-modal and multi-scale remote sensing data for semantic labeling. Our contributions are threefold: a) we present an efficient multi-scale approach to leverage both a large spatial context and the high resolution data, b) we investigate early and late fusion of Lidar and multispectral data, c) we validate our methods on two public datasets with state-of-the-art results. Our results indicate that late fusion make it possible to recover errors steaming from ambiguous data, while early fusion allows for better joint-feature learning but at the cost of higher sensitivity to missing data.