# Increasing the Robustness of Semantic Segmentation Models with Painting-by-Numbers

arXiv: https://arxiv.org/abs/2010.05495 · HTML: https://mlsanity.com/paper/2010.05495/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2020.

## Built on by (most cited first, 17 of 17)

- [Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2312.04265/) (2023)
- [Test-Time Adaptation to Distribution Shift by Confidence Maximization and Input Transformation](https://mlsanity.com/paper/2106.14999/) (2021)
- [SoMA: Singular Value Decomposed Minor Components Adaptation for Domain Generalizable Representation Learning](https://mlsanity.com/paper/2412.04077/) (2024)
- [Does Robustness on ImageNet Transfer to Downstream Tasks?](https://mlsanity.com/paper/2204.03934/) (2022)
- [Benchmarking the Robustness of Instance Segmentation Models](https://mlsanity.com/paper/2109.01123/) (2021)
- [StyleLess layer: Improving robustness for real-world driving](https://mlsanity.com/paper/2103.13905/) (2021)
- [VITA: A Multi-Source Vicinal Transfer Augmentation Method for Out-of-Distribution Generalization](https://mlsanity.com/paper/2204.11531/) (2022)
- [Towards Robust Semantic Segmentation against Patch-based Attack via Attention Refinement](https://mlsanity.com/paper/2401.01750/) (2024)
- [Rein++: Efficient Generalization and Adaptation for Semantic Segmentation with Vision Foundation Models](https://mlsanity.com/paper/2508.01667/) (2025)
- [Decomposing and Revising What Language Models Generate](https://mlsanity.com/paper/2509.00765/) (2025)
- [Quantifying the robustness of deep multispectral segmentation models against natural perturbations and data poisoning](https://mlsanity.com/paper/2305.11347/) (2023)
- [Exploring the Robustness of Human Parsers Towards Common Corruptions](https://mlsanity.com/paper/2309.00938/) (2023)
- [Artificial Intelligence for Geometry-Based Feature Extraction, Analysis and Synthesis in Artistic Images: A Survey](https://mlsanity.com/paper/2412.01450/) (2024)
- ["Just Drive": Colour Bias Mitigation for Semantic Segmentation in the Context of Urban Driving](https://mlsanity.com/paper/2112.01121/) (2021)
- [On the Influence of Shape, Texture and Color for Learning Semantic Segmentation](https://mlsanity.com/paper/2410.14878/) (2024)
- [Contrast All the Time: Learning Time Series Representation from Temporal Consistency](https://mlsanity.com/paper/2410.15416/) (2024)
- [RecycleLoRA: Rank-Revealing QR-Based Dual-LoRA Subspace Adaptation for Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2603.28142/) (2026)

## Abstract (arXiv)

For safety-critical applications such as autonomous driving, CNNs have to be robust with respect to unavoidable image corruptions, such as image noise. While previous works addressed the task of robust prediction in the context of full-image classification, we consider it for dense semantic segmentation. We build upon an insight from image classification that output robustness can be improved by increasing the network-bias towards object shapes. We present a new training schema that increases this shape bias. Our basic idea is to alpha-blend a portion of the RGB training images with faked images, where each class-label is given a fixed, randomly chosen color that is not likely to appear in real imagery. This forces the network to rely more strongly on shape cues. We call this data augmentation technique ``Painting-by-Numbers''. We demonstrate the effectiveness of our training schema for DeepLabv3+ with various network backbones, MobileNet-V2, ResNets, and Xception, and evaluate it on the Cityscapes dataset. With respect to our 16 different types of image corruptions and 5 different network backbones, we are in 74% better than training with clean data. For cases where we are worse than a model trained without our training schema, it is mostly only marginally worse. However, for some image corruptions such as images with noise, we see a considerable performance gain of up to 25%.