# Towards Reliable Evaluation and Fast Training of Robust Semantic Segmentation Models

arXiv: https://arxiv.org/abs/2306.12941 · HTML: https://mlsanity.com/paper/2306.12941/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 51 references that are in the map.
- First posted on arXiv 22 Jun 2023.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Evasion Attacks against Machine Learning at Test Time](https://mlsanity.com/paper/1708.06131/) (2017)
- [RobustBench: a standardized adversarial robustness benchmark](https://mlsanity.com/paper/2010.09670/) (2020)
- [On Adaptive Attacks to Adversarial Example Defenses](https://mlsanity.com/paper/2002.08347/) (2020)
- [Adversarial Patch](https://mlsanity.com/paper/1712.09665/) (2017)
- [Do Adversarially Robust ImageNet Models Transfer Better?](https://mlsanity.com/paper/2007.08489/) (2020)
- [Adversarial Examples for Semantic Segmentation and Object Detection](https://mlsanity.com/paper/1703.08603/) (2017)
- [Are Transformers More Robust Than CNNs?](https://mlsanity.com/paper/2111.05464/) (2021)
- [EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examples](https://mlsanity.com/paper/1709.04114/) (2017)
- [Perceptual Adversarial Robustness: Defense Against Unseen Threat Models](https://mlsanity.com/paper/2006.12655/) (2020)
- [On the Robustness of Semantic Segmentation Models to Adversarial Attacks](https://mlsanity.com/paper/1711.09856/) (2017)
- [Decoupling Direction and Norm for Efficient Gradient-Based L2 Adversarial Attacks and Defenses](https://mlsanity.com/paper/1811.09600/) (2018)
- [Universal Adversarial Perturbations Against Semantic Image Segmentation](https://mlsanity.com/paper/1704.05712/) (2017)
- [Wasserstein Adversarial Examples via Projected Sinkhorn Iterations](https://mlsanity.com/paper/1902.07906/) (2019)
- [Generalizable Data-free Objective for Crafting Universal Adversarial Perturbations](https://mlsanity.com/paper/1801.08092/) (2018)
- [A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking](https://mlsanity.com/paper/2302.14301/) (2023)

## Built on by (most cited first, 12 of 12)

- [Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs](https://mlsanity.com/paper/2509.02820/) (2025)
- [PatchGuard: Adversarially Robust Anomaly Detection and Localization through Vision Transformers and Pseudo Anomalies](https://mlsanity.com/paper/2506.09237/) (2025)
- [PartImageNet++ Dataset: Scaling up Part-based Models for Robust Recognition](https://mlsanity.com/paper/2407.10918/) (2024)
- [Segmentation-guided MRI reconstruction for meaningfully diverse reconstructions](https://mlsanity.com/paper/2407.18026/) (2024)
- [Segment (Almost) Nothing: Prompt-Agnostic Adversarial Attacks on Segmentation Models](https://mlsanity.com/paper/2311.14450/) (2023)
- [Red-Teaming Segment Anything Model](https://mlsanity.com/paper/2404.02067/) (2024)
- [Rethinking Semi-supervised Segmentation Beyond Accuracy: Reliability and Robustness](https://mlsanity.com/paper/2506.05917/) (2025)
- [On the Robustness of Adversarial Training Against Uncertainty Attacks](https://mlsanity.com/paper/2410.21952/) (2024)
- [Evaluating the Adversarial Robustness of Semantic Segmentation: Trying Harder Pays Off](https://mlsanity.com/paper/2407.09150/) (2024)
- [On the Adversarial Robustness of Discrete Image Tokenizers](https://mlsanity.com/paper/2602.18252/) (2026)
- [TsallisPGD: Adaptive Gradient Weighting for Adversarial Attacks on Semantic Segmentation](https://mlsanity.com/paper/2605.03405/) (2026)
- [SegPAR: Class-Centric Decision-Based Sparse Attack for Semantic Segmentation](https://mlsanity.com/paper/2608.11285/) (2026)

## Abstract (arXiv)

Adversarial robustness has been studied extensively in image classification, especially for the ℓ_∞-threat model, but significantly less so for related tasks such as object detection and semantic segmentation, where attacks turn out to be a much harder optimization problem than for image classification. We propose several problem-specific novel attacks minimizing different metrics in accuracy and mIoU. The ensemble of our attacks, SEA, shows that existing attacks severely overestimate the robustness of semantic segmentation models. Surprisingly, existing attempts of adversarial training for semantic segmentation models turn out to be weak or even completely non-robust. We investigate why previous adaptations of adversarial training to semantic segmentation failed and show how recently proposed robust ImageNet backbones can be used to obtain adversarially robust semantic segmentation models with up to six times less training time for PASCAL-VOC and the more challenging ADE20k. The associated code and robust models are available at https://github.com/nmndeep/robust-segmentation