# Automated Evaluation of Semantic Segmentation Robustness for Autonomous Driving

arXiv: https://arxiv.org/abs/1810.10193 · HTML: https://mlsanity.com/paper/1810.10193/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2018.

## Built on by (most cited first, 21 of 21)

- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [Panoptic Segmentation: A Review](https://mlsanity.com/paper/2111.10250/) (2021)
- [Unsupervised Foggy Scene Understanding via Self Spatial-Temporal Label Diffusion](https://mlsanity.com/paper/2206.04879/) (2022)
- [RaidaR: A Rich Annotated Image Dataset of Rainy Street Scenes](https://mlsanity.com/paper/2104.04606/) (2021)
- [ShareCMP: Polarization-Aware RGB-P Semantic Segmentation](https://mlsanity.com/paper/2312.03430/) (2023)
- [EpiTESTER: Testing Autonomous Vehicles with Epigenetic Algorithm and Attention Mechanism](https://mlsanity.com/paper/2312.00207/) (2023)
- [Compensating for Sensing Failures via Delegation in Human-AI Hybrid Systems](https://mlsanity.com/paper/2303.01300/) (2023)
- [ShadowHack: Hacking Shadows via Luminance-Color Divide and Conquer](https://mlsanity.com/paper/2412.02545/) (2024)
- [Deep Multi-Branch Aggregation Network for Real-Time Semantic Segmentation in Street Scenes](https://mlsanity.com/paper/2203.04037/) (2022)
- [DeepQTest: Testing Autonomous Driving Systems with Reinforcement Learning and Real-world Weather Data](https://mlsanity.com/paper/2310.05170/) (2023)
- [Optimizing Delegation in Collaborative Human-AI Hybrid Teams](https://mlsanity.com/paper/2402.05605/) (2024)
- [Safety Monitoring of Machine Learning Perception Functions: a Survey](https://mlsanity.com/paper/2412.06869/) (2024)
- [Rethinking Semi-supervised Segmentation Beyond Accuracy: Reliability and Robustness](https://mlsanity.com/paper/2506.05917/) (2025)
- [Improving Online Performance Prediction for Semantic Segmentation](https://mlsanity.com/paper/2104.05255/) (2021)
- [Evaluating Robustness over High Level Driving Instruction for Autonomous Driving](https://mlsanity.com/paper/2105.10014/) (2021)
- [Work In Progress: Safety and Robustness Verification of Autoencoder-Based Regression Models using the NNV Tool](https://mlsanity.com/paper/2207.06759/) (2022)
- [Global and Local Features through Gaussian Mixture Models on Image Semantic Segmentation](https://mlsanity.com/paper/2207.09162/) (2022)
- [Preventing Errors in Person Detection: A Part-Based Self-Monitoring Framework](https://mlsanity.com/paper/2307.04533/) (2023)
- [AD-SAM: Adapting the Segment Anything Model for Semantic Segmentation in Autonomous Driving](https://mlsanity.com/paper/2510.27047/) (2025)
- [WeatherSeg: Weather-Robust Image Segmentation using Teacher-Student Dual Learning and Classifier-Updating Attention](https://mlsanity.com/paper/2604.22824/) (2026)
- [The Impact of CutMix on Reliability and Robustness in Semantic Segmentation](https://mlsanity.com/paper/2608.18715/) (2026)

## Abstract (arXiv)

One of the fundamental challenges in the design of perception systems for autonomous vehicles is validating the performance of each algorithm under a comprehensive variety of operating conditions. In the case of vision-based semantic segmentation, there are known issues when encountering new scenarios that are sufficiently different to the training data. In addition, even small variations in environmental conditions such as illumination and precipitation can affect the classification performance of the segmentation model. Given the reliance on visual information, these effects often translate into poor semantic pixel classification which can potentially lead to catastrophic consequences when driving autonomously. This paper presents a novel method for analysing the robustness of semantic segmentation models and provides a number of metrics to evaluate the classification performance over a variety of environmental conditions. The process incorporates an additional sensor (lidar) to automate the process, eliminating the need for labour-intensive hand labelling of validation data. The system integrity can be monitored as the performance of the vision sensors are validated against a different sensor modality. This is necessary for detecting failures that are inherent to vision technology. Experimental results are presented based on multiple datasets collected at different times of the year with different environmental conditions. These results show that the semantic segmentation performance varies depending on the weather, camera parameters, existence of shadows, etc.. The results also demonstrate how the metrics can be used to compare and validate the performance after making improvements to a model, and compare the performance of different networks.