# Addressing Mistake Severity in Neural Networks with Semantic Knowledge

arXiv: https://arxiv.org/abs/2211.11880 · HTML: https://mlsanity.com/paper/2211.11880/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 20 references that are in the map.
- First posted on arXiv 21 Nov 2022.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Spatially Transformed Adversarial Examples](https://mlsanity.com/paper/1801.02612/) (2018)
- [Making Better Mistakes: Leveraging Class Hierarchies with Deep Networks](https://mlsanity.com/paper/1912.09393/) (2019)
- [Semantic Adversarial Examples](https://mlsanity.com/paper/1804.00499/) (2018)
- [Improving robustness against common corruptions with frequency biased models](https://mlsanity.com/paper/2103.16241/) (2021)
- [Attribute-Guided Adversarial Training for Robustness to Natural Perturbations](https://mlsanity.com/paper/2012.01806/) (2020)
- [Adversarial Robustness as a Prior for Learned Representations](https://mlsanity.com/paper/1906.00945/) (2019)
- [Demiguise Attack: Crafting Invisible Semantic Adversarial Perturbations with Perceptual Similarity](https://mlsanity.com/paper/2107.01396/) (2021)
- [Image Synthesis with a Single (Robust) Classifier](https://mlsanity.com/paper/1906.09453/) (2019)
- [Tools and Practices for Responsible AI Engineering](https://mlsanity.com/paper/2201.05647/) (2022)
- [Improving Hierarchical Adversarial Robustness of Deep Neural Networks](https://mlsanity.com/paper/2102.09012/) (2021)
- [Robust Training Using Natural Transformation](https://mlsanity.com/paper/2105.04070/) (2021)

## Built on by (most cited first, 3 of 3)

- [PosterSum: A Multimodal Benchmark for Scientific Poster Summarization](https://mlsanity.com/paper/2502.17540/) (2025)
- [Addressing Discrepancies in Semantic and Visual Alignment in Neural Networks](https://mlsanity.com/paper/2306.01148/) (2023)
- [Hier-COS: Making Deep Features Hierarchy-aware via Composition of Orthogonal Subspaces](https://mlsanity.com/paper/2503.07853/) (2025)

## Abstract (arXiv)

Robustness in deep neural networks and machine learning algorithms in general is an open research challenge. In particular, it is difficult to ensure algorithmic performance is maintained on out-of-distribution inputs or anomalous instances that cannot be anticipated at training time. Embodied agents will be deployed in these conditions, and are likely to make incorrect predictions. An agent will be viewed as untrustworthy unless it can maintain its performance in dynamic environments. Most robust training techniques aim to improve model accuracy on perturbed inputs; as an alternate form of robustness, we aim to reduce the severity of mistakes made by neural networks in challenging conditions. We leverage current adversarial training methods to generate targeted adversarial attacks during the training process in order to increase the semantic similarity between a model's predictions and true labels of misclassified instances. Results demonstrate that our approach performs better with respect to mistake severity compared to standard and adversarially trained models. We also find an intriguing role that non-robust features play with regards to semantic similarity.