# Analyzing and Improving Neural Networks by Generating Semantic Counterexamples through Differentiable Rendering

arXiv: https://arxiv.org/abs/1910.00727 · HTML: https://mlsanity.com/paper/1910.00727/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2019.

## Built on by (most cited first, 2 of 2)

- [Exposing Previously Undetectable Faults in Deep Neural Networks](https://mlsanity.com/paper/2106.00576/) (2021)
- [In-distribution adversarial attacks on object recognition models using gradient-free search](https://mlsanity.com/paper/2106.16198/) (2021)

## Abstract (arXiv)

Even as deep neural networks (DNNs) have achieved remarkable success on vision-related tasks, their performance is brittle to transformations in the input. Of particular interest are semantic transformations that model changes that have a basis in the physical world, such as rotations, translations, changes in lighting or camera pose. In this paper, we show how differentiable rendering can be utilized to generate images that are informative, yet realistic, and which can be used to analyze DNN performance and improve its robustness through data augmentation. Given a differentiable renderer and a DNN, we show how to use off-the-shelf attacks from adversarial machine learning to generate semantic counterexamples -- images where semantic features are changed as to produce misclassifications or misdetections. We validate our approach on DNNs for image classification and object detection. For classification, we show that semantic counterexamples, when used to augment the dataset, (i) improve generalization performance (ii) enhance robustness to semantic transformations, and (iii) transfer between models. Additionally, in comparison to sampling-based semantic augmentation, our technique generates more informative data in a sample efficient manner.