# Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather

arXiv: https://arxiv.org/abs/1902.08913 · HTML: https://mlsanity.com/paper/1902.08913/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2019.

## Built on by (most cited first, 9 of 9)

- [3D and 4D World Modeling: A Survey](https://mlsanity.com/paper/2509.07996/) (2025)
- [Federated Learning in Intelligent Transportation Systems: Recent Applications and Open Problems](https://mlsanity.com/paper/2309.11039/) (2023)
- [Vision-RADAR fusion for Robotics BEV Detections: A Survey](https://mlsanity.com/paper/2302.06643/) (2023)
- [Graph-based Thermal-Inertial SLAM with Probabilistic Neural Networks](https://mlsanity.com/paper/2104.07196/) (2021)
- [An Unsupervised Domain Adaptive Approach for Multimodal 2D Object Detection in Adverse Weather Conditions](https://mlsanity.com/paper/2203.03568/) (2022)
- [Deep learning for radar data exploitation of autonomous vehicle](https://mlsanity.com/paper/2203.08038/) (2022)
- [t-READi: Transformer-Powered Robust and Efficient Multimodal Inference for Autonomous Driving](https://mlsanity.com/paper/2410.09747/) (2024)
- [PAN: Pillars-Attention-Based Network for 3D Object Detection](https://mlsanity.com/paper/2509.15935/) (2025)
- [ModalPatch: A Plug-and-Play Module for Robust Multi-Modal 3D Object Detection under Modality Drop](https://mlsanity.com/paper/2603.02481/) (2026)

## Abstract (arXiv)

The fusion of multimodal sensor streams, such as camera, lidar, and radar measurements, plays a critical role in object detection for autonomous vehicles, which base their decision making on these inputs. While existing methods exploit redundant information in good environmental conditions, they fail in adverse weather where the sensory streams can be asymmetrically distorted. These rare "edge-case" scenarios are not represented in available datasets, and existing fusion architectures are not designed to handle them. To address this challenge we present a novel multimodal dataset acquired in over 10,000km of driving in northern Europe. Although this dataset is the first large multimodal dataset in adverse weather, with 100k labels for lidar, camera, radar, and gated NIR sensors, it does not facilitate training as extreme weather is rare. To this end, we present a deep fusion network for robust fusion without a large corpus of labeled training data covering all asymmetric distortions. Departing from proposal-level fusion, we propose a single-shot model that adaptively fuses features, driven by measurement entropy. We validate the proposed method, trained on clean data, on our extensive validation dataset. Code and data are available here https://github.com/princeton-computational-imaging/SeeingThroughFog.