# Improving Semantic Segmentation via Video Propagation and Label Relaxation

arXiv: https://arxiv.org/abs/1812.01593 · HTML: https://mlsanity.com/paper/1812.01593/

## Key facts

- Cited by 134 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Dec 2018.

## Built on by (most cited first, 30 of 134)

- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021)
- [RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening](https://mlsanity.com/paper/2103.15597/) (2021)
- [SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation](https://mlsanity.com/paper/2104.14812/) (2021)
- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [Pixel-wise Anomaly Detection in Complex Driving Scenes](https://mlsanity.com/paper/2103.05445/) (2021)
- [Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes](https://mlsanity.com/paper/2111.12264/) (2021)
- [Standardized Max Logits: A Simple yet Effective Approach for Identifying Unexpected Road Obstacles in Urban-Scene Segmentation](https://mlsanity.com/paper/2107.11264/) (2021)
- [Fine-grained Semantics-aware Representation Enhancement for Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2108.08829/) (2021)
- [Residual Pattern Learning for Pixel-wise Out-of-Distribution Detection in Semantic Segmentation](https://mlsanity.com/paper/2211.14512/) (2022)
- [A Dynamic Multi-Scale Voxel Flow Network for Video Prediction](https://mlsanity.com/paper/2303.09875/) (2023)
- [Towards Robust Rain Removal Against Adversarial Attacks: A Comprehensive Benchmark Analysis and Beyond](https://mlsanity.com/paper/2203.16931/) (2022)
- [DenseHybrid: Hybrid Anomaly Detection for Dense Open-set Recognition](https://mlsanity.com/paper/2207.02606/) (2022)
- [Active Boundary Loss for Semantic Segmentation](https://mlsanity.com/paper/2102.02696/) (2021)
- [Backdoor Attacks Against Deep Image Compression via Adaptive Frequency Trigger](https://mlsanity.com/paper/2302.14677/) (2023)
- [Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety](https://mlsanity.com/paper/2104.14235/) (2021)
- [TubeFormer-DeepLab: Video Mask Transformer](https://mlsanity.com/paper/2205.15361/) (2022)
- [Maskomaly:Zero-Shot Mask Anomaly Segmentation](https://mlsanity.com/paper/2305.16972/) (2023)
- [AuxAdapt: Stable and Efficient Test-Time Adaptation for Temporally Consistent Video Semantic Segmentation](https://mlsanity.com/paper/2110.12369/) (2021)
- [MGNet: Monocular Geometric Scene Understanding for Autonomous Driving](https://mlsanity.com/paper/2206.13199/) (2022)
- [Mining Relations among Cross-Frame Affinities for Video Semantic Segmentation](https://mlsanity.com/paper/2207.10436/) (2022)
- [Learning Local and Global Temporal Contexts for Video Semantic Segmentation](https://mlsanity.com/paper/2204.03330/) (2022)
- [Video-kMaX: A Simple Unified Approach for Online and Near-Online Video Panoptic Segmentation](https://mlsanity.com/paper/2304.04694/) (2023)
- [S2R-DepthNet: Learning a Generalizable Depth-specific Structural Representation](https://mlsanity.com/paper/2104.00877/) (2021)
- [Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction](https://mlsanity.com/paper/2104.06697/) (2021)
- [Domain Adaptive Video Segmentation via Temporal Consistency Regularization](https://mlsanity.com/paper/2107.11004/) (2021)
- [NightLab: A Dual-level Architecture with Hardness Detection for Segmentation at Night](https://mlsanity.com/paper/2204.05538/) (2022)
- [Automated Detection of Label Errors in Semantic Segmentation Datasets via Deep Learning and Uncertainty Quantification](https://mlsanity.com/paper/2207.06104/) (2022)
- [Self-Supervised Monocular Depth Estimation: Solving the Edge-Fattening Problem](https://mlsanity.com/paper/2210.00411/) (2022)

## Abstract (arXiv)

Semantic segmentation requires large amounts of pixel-wise annotations to learn accurate models. In this paper, we present a video prediction-based methodology to scale up training sets by synthesizing new training samples in order to improve the accuracy of semantic segmentation networks. We exploit video prediction models' ability to predict future frames in order to also predict future labels. A joint propagation strategy is also proposed to alleviate mis-alignments in synthesized samples. We demonstrate that training segmentation models on datasets augmented by the synthesized samples leads to significant improvements in accuracy. Furthermore, we introduce a novel boundary label relaxation technique that makes training robust to annotation noise and propagation artifacts along object boundaries. Our proposed methods achieve state-of-the-art mIoUs of 83.5% on Cityscapes and 82.9% on CamVid. Our single model, without model ensembles, achieves 72.8% mIoU on the KITTI semantic segmentation test set, which surpasses the winning entry of the ROB challenge 2018. Our code and videos can be found at https://nv-adlr.github.io/publication/2018-Segmentation.