# DGFusion: Depth-Guided Sensor Fusion for Robust Semantic Perception

arXiv: https://arxiv.org/abs/2509.09828 · HTML: https://mlsanity.com/paper/2509.09828/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 11 Sep 2025.

## Builds on (most cited first, 27 of 27)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [UniDepth: Universal Monocular Metric Depth Estimation](https://mlsanity.com/paper/2403.18913/) (2024)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)
- [OneFormer: One Transformer to Rule Universal Image Segmentation](https://mlsanity.com/paper/2211.06220/) (2022)
- [UniDepthV2: Universal Monocular Metric Depth Estimation Made Simpler](https://mlsanity.com/paper/2502.20110/) (2025)
- [Delivering Arbitrary-Modal Semantic Segmentation](https://mlsanity.com/paper/2303.01480/) (2023)
- [Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather](https://mlsanity.com/paper/2108.05249/) (2021)
- [LiDAR Snowfall Simulation for Robust 3D Object Detection](https://mlsanity.com/paper/2203.15118/) (2022)
- [The Edge of Depth: Explicit Constraints between Segmentation and Depth](https://mlsanity.com/paper/2004.00171/) (2020)
- [Exploring Relational Context for Multi-Task Dense Prediction](https://mlsanity.com/paper/2104.13874/) (2021)
- [Does computer vision matter for action?](https://mlsanity.com/paper/1905.12887/) (2019)
- [GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer](https://mlsanity.com/paper/2406.01210/) (2024)
- [StitchFusion: Weaving Any Visual Modalities to Enhance Multimodal Semantic Segmentation](https://mlsanity.com/paper/2408.01343/) (2024)
- [MUSES: The Multi-Sensor Semantic Perception Dataset for Driving under Uncertainty](https://mlsanity.com/paper/2401.12761/) (2024)
- [CAFuser: Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenes](https://mlsanity.com/paper/2410.10791/) (2024)
- [HRFuser: A Multi-resolution Sensor Fusion Architecture for 2D Object Detection](https://mlsanity.com/paper/2206.15157/) (2022)
- [Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalities](https://mlsanity.com/paper/2407.11351/) (2024)
- [Centering the Value of Every Modality: Towards Efficient and Resilient Modality-agnostic Semantic Segmentation](https://mlsanity.com/paper/2407.11344/) (2024)
- [Contrastive Model Adaptation for Cross-Condition Robustness in Semantic Segmentation](https://mlsanity.com/paper/2303.05194/) (2023)
- [Pay "Attention" to Adverse Weather: Weather-aware Attention-based Object Detection](https://mlsanity.com/paper/2204.10803/) (2022)
- [Semi-Supervised Semantic Depth Estimation using Symbiotic Transformer and NearFarMix Augmentation](https://mlsanity.com/paper/2308.14400/) (2023)

## Built on by (most cited first, 4 of 4)

- [Geometry-Aware Cross Modal Alignment for Light Field-LiDAR Semantic Segmentation](https://mlsanity.com/paper/2510.06687/) (2025)
- [MT-Depth: Multi-task Instance feature analysis for the Depth Completion](https://mlsanity.com/paper/2512.04734/) (2025)
- [Adverse-to-the-eXtreme Panoptic Segmentation: URVIS 2026 Study and Benchmark](https://mlsanity.com/paper/2604.16984/) (2026)
- [DVPSFormer: Efficient Online Depth-aware Video Panoptic Segmentation for Autonomous Driving](https://mlsanity.com/paper/2607.26165/) (2026)

## Abstract (arXiv)

Robust semantic perception for autonomous vehicles relies on effectively combining multiple sensors with complementary strengths and weaknesses. State-of-the-art sensor fusion approaches to semantic perception often treat sensor data uniformly across the spatial extent of the input, which hinders performance when faced with challenging conditions. By contrast, we propose a novel depth-guided multimodal fusion method that upgrades condition-aware fusion by integrating depth information. Our network, DGFusion, poses multimodal segmentation as a multi-task problem, utilizing the lidar measurements, which are typically available in outdoor sensor suites, both as one of the model's inputs and as ground truth for learning depth. Our corresponding auxiliary depth head helps to learn depth-aware features, which are encoded into spatially varying local depth tokens that condition our attentive cross-modal fusion. Together with a global condition token, these local depth tokens dynamically adapt sensor fusion to the spatially varying reliability of each sensor across the scene, which largely depends on depth. In addition, we propose a robust loss for our depth, which is essential for learning from lidar inputs that are typically sparse and noisy in adverse conditions. Our method achieves state-of-the-art panoptic and semantic segmentation performance on the challenging MUSES and DeLiVER datasets. Code and models are available at https://github.com/timbroed/DGFusion