# Unsupervised Foggy Scene Understanding via Self Spatial-Temporal Label Diffusion

arXiv: https://arxiv.org/abs/2206.04879 · HTML: https://mlsanity.com/paper/2206.04879/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 59 references that are in the map.
- First posted on arXiv 10 Jun 2022.

## Builds on (most cited first, 30 of 37)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Deep Transfer Learning with Joint Adaptation Networks](https://mlsanity.com/paper/1605.06636/) (2016)
- [Learning to Adapt Structured Output Space for Semantic Segmentation](https://mlsanity.com/paper/1802.10349/) (2018)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [Confidence Regularized Self-Training](https://mlsanity.com/paper/1908.09822/) (2019)
- [Contrastive Adaptation Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1901.00976/) (2019)
- [Semantic Foggy Scene Understanding with Synthetic Data](https://mlsanity.com/paper/1708.07819/) (2017)
- [Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform](https://mlsanity.com/paper/1804.02815/) (2018)
- [Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks](https://mlsanity.com/paper/1612.05424/) (2016)
- [Bidirectional Learning for Domain Adaptation of Semantic Segmentation](https://mlsanity.com/paper/1904.10620/) (2019)
- [Instance Adaptive Self-Training for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2008.12197/) (2020)
- [Image to Image Translation for Domain Adaptation](https://mlsanity.com/paper/1712.00479/) (2017)
- [Learning Texture Invariant Representation for Domain Adaptation of Semantic Segmentation](https://mlsanity.com/paper/2003.00867/) (2020)
- [Deep Semantic Segmentation of Natural and Medical Images: A Review](https://mlsanity.com/paper/1910.07655/) (2019)
- [Learning Accurate Dense Correspondences and When to Trust Them](https://mlsanity.com/paper/2101.01710/) (2021)
- [Constructing Self-motivated Pyramid Curriculums for Cross-Domain Semantic Segmentation: A Non-Adversarial Approach](https://mlsanity.com/paper/1908.09547/) (2019)
- [Model Adaptation with Synthetic and Real Data for Semantic Dense Foggy Scene Understanding](https://mlsanity.com/paper/1808.01265/) (2018)
- [Automatic adaptation of object detectors to new domains using self-training](https://mlsanity.com/paper/1904.07305/) (2019)
- [Squeeze-and-Attention Networks for Semantic Segmentation](https://mlsanity.com/paper/1909.03402/) (2019)
- [Two-phase Pseudo Label Densification for Self-training based Domain Adaptation](https://mlsanity.com/paper/2012.04828/) (2020)
- [A Curriculum Domain Adaptation Approach to the Semantic Segmentation of Urban Scenes](https://mlsanity.com/paper/1812.09953/) (2018)
- [Guidance and Evaluation: Semantic-Aware Image Inpainting for Mixed Scenes](https://mlsanity.com/paper/2003.06877/) (2020)
- [Curriculum Model Adaptation with Synthetic and Real Data for Semantic Foggy Scene Understanding](https://mlsanity.com/paper/1901.01415/) (2019)
- [Improving Object Detection with Selective Self-supervised Self-training](https://mlsanity.com/paper/2007.09162/) (2020)
- [Affinity Space Adaptation for Semantic Segmentation Across Domains](https://mlsanity.com/paper/2009.12559/) (2020)

## Built on by (most cited first, 8 of 8)

- [Refign: Align and Refine for Adaptation of Semantic Segmentation to Adverse Conditions](https://mlsanity.com/paper/2207.06825/) (2022)
- [Low-light Image Enhancement via CLIP-Fourier Guided Wavelet Diffusion](https://mlsanity.com/paper/2401.03788/) (2024)
- [Refined Semantic Enhancement towards Frequency Diffusion for Video Captioning](https://mlsanity.com/paper/2211.15076/) (2022)
- [Magic ELF: Image Deraining Meets Association Learning and Transformer](https://mlsanity.com/paper/2207.10455/) (2022)
- [Open Scene Understanding: Grounded Situation Recognition Meets Segment Anything for Helping People with Visual Impairments](https://mlsanity.com/paper/2307.07757/) (2023)
- [EQ-TAA: Equivariant Traffic Accident Anticipation via Diffusion-Based Accident Video Synthesis](https://mlsanity.com/paper/2506.10002/) (2025)
- [Dynamic Association Learning of Self-Attention and Convolution in Image Restoration](https://mlsanity.com/paper/2311.05147/) (2023)
- [DAWA: Dynamic Ambiguity-Wise Adaptation for Real-Time Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2409.01072/) (2024)

## Abstract (arXiv)

Understanding foggy image sequence in the driving scenes is critical for autonomous driving, but it remains a challenging task due to the difficulty in collecting and annotating real-world images of adverse weather. Recently, the self-training strategy has been considered a powerful solution for unsupervised domain adaptation, which iteratively adapts the model from the source domain to the target domain by generating target pseudo labels and re-training the model. However, the selection of confident pseudo labels inevitably suffers from the conflict between sparsity and accuracy, both of which will lead to suboptimal models. To tackle this problem, we exploit the characteristics of the foggy image sequence of driving scenes to densify the confident pseudo labels. Specifically, based on the two discoveries of local spatial similarity and adjacent temporal correspondence of the sequential image data, we propose a novel Target-Domain driven pseudo label Diffusion (TDo-Dif) scheme. It employs superpixels and optical flows to identify the spatial similarity and temporal correspondence, respectively and then diffuses the confident but sparse pseudo labels within a superpixel or a temporal corresponding pair linked by the flow. Moreover, to ensure the feature similarity of the diffused pixels, we introduce local spatial similarity loss and temporal contrastive loss in the model re-training stage. Experimental results show that our TDo-Dif scheme helps the adaptive model achieve 51.92% and 53.84% mean intersection-over-union (mIoU) on two publicly available natural foggy datasets (Foggy Zurich and Foggy Driving), which exceeds the state-of-the-art unsupervised domain adaptive semantic segmentation methods. Models and data can be found at https://github.com/velor2012/TDo-Dif.