# WeatherProof: A Paired-Dataset Approach to Semantic Segmentation in Adverse Weather

arXiv: https://arxiv.org/abs/2312.09534 · HTML: https://mlsanity.com/paper/2312.09534/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 15 Dec 2023.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](https://mlsanity.com/paper/2211.05778/) (2022)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [Semantic Foggy Scene Understanding with Synthetic Data](https://mlsanity.com/paper/1708.07819/) (2017)
- [ICNet for Real-Time Semantic Segmentation on High-Resolution Images](https://mlsanity.com/paper/1704.08545/) (2017)
- [Density-aware Single Image De-raining using a Multi-stream Dense Network](https://mlsanity.com/paper/1802.07412/) (2018)
- [TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions](https://mlsanity.com/paper/2111.14813/) (2021)
- [DesnowNet: Context-Aware Deep Network for Snow Removal](https://mlsanity.com/paper/1708.04512/) (2017)
- [Spatial Attentive Single-Image Deraining with a High Quality Real Rain Dataset](https://mlsanity.com/paper/1904.01538/) (2019)
- [Heavy Rain Image Restoration: Integrating Physics Model and Conditional Adversarial Learning](https://mlsanity.com/paper/1904.05050/) (2019)
- [A Model-driven Deep Neural Network for Single Image Rain Removal](https://mlsanity.com/paper/2005.01333/) (2020)
- [Uncertainty Guided Multi-Scale Residual Learning-using a Cycle Spinning CNN for Single Image De-Raining](https://mlsanity.com/paper/1906.11129/) (2019)
- [Learning to Segment Every Thing](https://mlsanity.com/paper/1711.10370/) (2017)
- [Real-time Semantic Segmentation of Crop and Weed for Precision Agriculture Robots Leveraging Background Knowledge in CNNs](https://mlsanity.com/paper/1709.06764/) (2017)
- [ISTR: End-to-End Instance Segmentation with Transformers](https://mlsanity.com/paper/2105.00637/) (2021)
- [Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations](https://mlsanity.com/paper/1809.04766/) (2018)
- [Non-locally Enhanced Encoder-Decoder Network for Single Image De-raining](https://mlsanity.com/paper/1808.01491/) (2018)
- [Not Just Streaks: Towards Ground Truth for Single Image Deraining](https://mlsanity.com/paper/2206.10779/) (2022)
- [Bonnet: An Open-Source Training and Deployment Framework for Semantic Segmentation in Robotics using CNNs](https://mlsanity.com/paper/1802.08960/) (2018)
- [RTSeg: Real-time Semantic Segmentation Comparative Study](https://mlsanity.com/paper/1803.02758/) (2018)
- [Semantic Segmentation under Adverse Conditions: A Weather and Nighttime-aware Synthetic Data-based Approach](https://mlsanity.com/paper/2210.05626/) (2022)

## Built on by (most cited first, 9 of 9)

- [Detection-Friendly Nonuniformity Correction: A Union Framework for Infrared UAVTarget Detection](https://mlsanity.com/paper/2504.04012/) (2025)
- [Zero-Shot Video Deraining with Video Diffusion Models](https://mlsanity.com/paper/2511.18537/) (2025)
- [RobuMTL: Enhancing Multi-Task Learning Robustness Against Weather Conditions](https://mlsanity.com/paper/2601.10921/) (2026)
- [Enhanced Semantic Segmentation Pipeline for WeatherProof Dataset Challenge](https://mlsanity.com/paper/2406.03799/) (2024)
- [A Two-Stage Adverse Weather Semantic Segmentation Method for WeatherProof Challenge CVPR 2024 Workshop UG2+](https://mlsanity.com/paper/2406.05513/) (2024)
- [Solution for CVPR 2024 UG2+ Challenge Track on All Weather Semantic Segmentation](https://mlsanity.com/paper/2406.05837/) (2024)
- [Technical Report for CVPR 2024 WeatherProof Dataset Challenge: Semantic Segmentation on Paired Real Data](https://mlsanity.com/paper/2407.01579/) (2024)
- [A Robust Semantic Segmentation Pipeline for the CVPR 2026 8th UG2+ Challenge Track 2](https://mlsanity.com/paper/2605.22216/) (2026)
- [PairedGTA: Generating Driving Datasets for Controlled Photometric Shift Analysis](https://mlsanity.com/paper/2606.01192/) (2026)

## Abstract (arXiv)

The introduction of large, foundational models to computer vision has led to drastically improved performance on the task of semantic segmentation. However, these existing methods exhibit a large performance drop when testing on images degraded by weather conditions such as rain, fog, or snow. We introduce a general paired-training method that can be applied to all current foundational model architectures that leads to improved performance on images in adverse weather conditions. To this end, we create the WeatherProof Dataset, the first semantic segmentation dataset with accurate clear and adverse weather image pairs, which not only enables our new training paradigm, but also improves the evaluation of the performance gap between clear and degraded segmentation. We find that training on these paired clear and adverse weather frames which share an underlying scene results in improved performance on adverse weather data. With this knowledge, we propose a training pipeline which accentuates the advantages of paired-data training using consistency losses and language guidance, which leads to performance improvements by up to 18.4% as compared to standard training procedures.