# Modeling Weather Uncertainty for Multi-weather Co-Presence Estimation

arXiv: https://arxiv.org/abs/2403.20092 · HTML: https://mlsanity.com/paper/2403.20092/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 67 references that are in the map.
- First posted on arXiv 29 Mar 2024.

## Builds on (most cited first, 30 of 33)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data](https://mlsanity.com/paper/2002.11297/) (2020)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening](https://mlsanity.com/paper/2103.15597/) (2021)
- [Geometry Uncertainty Projection Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2107.13774/) (2021)
- [General Multi-label Image Classification with Transformers](https://mlsanity.com/paper/2011.14027/) (2020)
- [Uncertainty-aware Joint Salient Object and Camouflaged Object Detection](https://mlsanity.com/paper/2104.02628/) (2021)
- [Striking the Right Balance with Uncertainty](https://mlsanity.com/paper/1901.07590/) (2019)
- [Uncertainty-Aware Unsupervised Domain Adaptation in Object Detection](https://mlsanity.com/paper/2103.00236/) (2021)
- [An Efficient Domain-Incremental Learning Approach to Drive in All Weather Conditions](https://mlsanity.com/paper/2204.08817/) (2022)
- [Residual Attention: A Simple but Effective Method for Multi-Label Recognition](https://mlsanity.com/paper/2108.02456/) (2021)
- [Ithaca365: Dataset and Driving Perception under Repeated and Challenging Weather Conditions](https://mlsanity.com/paper/2208.01166/) (2022)
- [Uncertainty-Aware Consistency Regularization for Cross-Domain Semantic Segmentation](https://mlsanity.com/paper/2004.08878/) (2020)
- [Transformer-based Dual Relation Graph for Multi-label Image Recognition](https://mlsanity.com/paper/2110.04722/) (2021)
- [Curriculum Model Adaptation with Synthetic and Real Data for Semantic Foggy Scene Understanding](https://mlsanity.com/paper/1901.01415/) (2019)
- [MonoLoco: Monocular 3D Pedestrian Localization and Uncertainty Estimation](https://mlsanity.com/paper/1906.06059/) (2019)
- [Evaluating Robustness of Predictive Uncertainty Estimation: Are Dirichlet-based Models Reliable?](https://mlsanity.com/paper/2010.14986/) (2020)
- [Joint Air Quality and Weather Prediction Based on Multi-Adversarial Spatiotemporal Networks](https://mlsanity.com/paper/2012.15037/) (2020)
- [Learning Content-enhanced Mask Transformer for Domain Generalized Urban-Scene Segmentation](https://mlsanity.com/paper/2307.00371/) (2023)
- [Both Style and Fog Matter: Cumulative Domain Adaptation for Semantic Foggy Scene Understanding](https://mlsanity.com/paper/2112.00484/) (2021)
- [DRT: A Lightweight Single Image Deraining Recursive Transformer](https://mlsanity.com/paper/2204.11385/) (2022)
- [Haze Visibility Enhancement: A Survey and Quantitative Benchmarking](https://mlsanity.com/paper/1607.06235/) (2016)

## Abstract (arXiv)

Images from outdoor scenes may be taken under various weather conditions. It is well studied that weather impacts the performance of computer vision algorithms and needs to be handled properly. However, existing algorithms model weather condition as a discrete status and estimate it using multi-label classification. The fact is that, physically, specifically in meteorology, weather are modeled as a continuous and transitional status. Instead of directly implementing hard classification as existing multi-weather classification methods do, we consider the physical formulation of multi-weather conditions and model the impact of physical-related parameter on learning from the image appearance. In this paper, we start with solid revisit of the physics definition of weather and how it can be described as a continuous machine learning and computer vision task. Namely, we propose to model the weather uncertainty, where the level of probability and co-existence of multiple weather conditions are both considered. A Gaussian mixture model is used to encapsulate the weather uncertainty and a uncertainty-aware multi-weather learning scheme is proposed based on prior-posterior learning. A novel multi-weather co-presence estimation transformer (MeFormer) is proposed. In addition, a new multi-weather co-presence estimation (MePe) dataset, along with 14 fine-grained weather categories and 16,078 samples, is proposed to benchmark both conventional multi-label weather classification task and multi-weather co-presence estimation task. Large scale experiments show that the proposed method achieves state-of-the-art performance and substantial generalization capabilities on both the conventional multi-label weather classification task and the proposed multi-weather co-presence estimation task. Besides, modeling weather uncertainty also benefits adverse-weather semantic segmentation.