Training Image Derivatives: Increased Accuracy and Universal Robustness
Not on the map. Why?
Abstract (arXiv)
Derivative training is an established method that can significantly increase the accuracy of neural networks in certain low-dimensional tasks. In this paper, we extend this improvement to an illustrative image analysis problem: reconstructing the vertices of a cube from its image. By training the derivatives with respect to the cube's six degrees of freedom, we achieve a 25-fold increase in accuracy for noiseless inputs. Additionally, derivative knowledge offers a novel approach to enhancing network robustness, which has traditionally been understood in terms of two types of vulnerabilities: excessive sensitivity to minor perturbations and failure to detect significant image changes. Conventional robust training relies on output invariance, which inherently creates a trade-off between these two vulnerabilities. By leveraging derivative information we compute non-trivial output changes in response to arbitrary input perturbations. This resolves the trade-off, yielding a network that is twice as robust and five times more accurate than the best case under the invariance assumption. Unlike conventional robust training, this outcome can be further improved by simply increasing the network capacity. This approach is applicable to phase retrieval problems and other scenarios where a sufficiently smooth manifold parametrization can be obtained.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 48 of its 99 references that are in the map. First posted on arXiv 21 Oct 2023.
Builds on
48 of 99 references.
References · 48
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2013Intriguing properties of neural networkscited by 5,155
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
- 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
- 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
- 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 920
- 2018Robustness May Be at Odds with Accuracycited by 880
- 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 279
- 2017Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradientscited by 236
- 2018Disentangling Adversarial Robustness and Generalizationcited by 172
36 more
- 2018Is Robustness the Cost of Accuracy? -- A Comprehensive Study on the Robustness of 18 Deep Image Classification Modelscited by 135
- 2019Robust Learning with Jacobian Regularizationcited by 129
- 2017Stabilizing Training of Generative Adversarial Networks through Regularizationcited by 118
- 2019Adversarial Examples Are a Natural Consequence of Test Error in Noisecited by 114
- 2019Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentationcited by 105
- 2018Towards the first adversarially robust neural network model on MNISTcited by 104
- 2018Improving DNN Robustness to Adversarial Attacks using Jacobian Regularizationcited by 88
- 2021On the effectiveness of adversarial training against common corruptionscited by 71
- 2016A Boundary Tilting Persepective on the Phenomenon of Adversarial Examplescited by 70
- 2018Excessive Invariance Causes Adversarial Vulnerabilitycited by 66
- 2020Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbationscited by 55
- 2017Efficient Defenses Against Adversarial Attackscited by 53
- 2021Score-Based Generative Classifierscited by 53
- 2020Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inferencecited by 45
- 2022A Survey of Robust Adversarial Training in Pattern Recognition: Fundamental, Theory, and Methodologiescited by 44
- 2015A Unified Gradient Regularization Family for Adversarial Examplescited by 43
- 2019Are Perceptually-Aligned Gradients a General Property of Robust Classifiers?cited by 43
- 2019Jacobian Adversarially Regularized Networks for Robustnesscited by 40
- 2020Provable tradeoffs in adversarially robust classificationcited by 40
- 2019Adversarial Robustness May Be at Odds With Simplicitycited by 39
- 2019Understanding the Limitations of Conditional Generative Modelscited by 28
- 2022Explicit Tradeoffs between Adversarial and Natural Distributional Robustnesscited by 20
- 2021Shift Invariance Can Reduce Adversarial Robustnesscited by 16
- 2017Global versus Localized Generative Adversarial Netscited by 12
- 2018Tangent-Normal Adversarial Regularization for Semi-supervised Learningcited by 10
- 2019Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustnesscited by 9
- 2020Can we have it all? On the Trade-off between Spatial and Adversarial Robustness of Neural Networkscited by 9
- 2020Adversarial Concurrent Training: Optimizing Robustness and Accuracy Trade-off of Deep Neural Networkscited by 9
- 2017Neural networks catching up with finite differences in solving partial differential equations in higher dimensionscited by 6
- 2022Sobolev Training for Implicit Neural Representations with Approximated Image Derivativescited by 6
- 2018Semi-Supervised Learning with GANs: Revisiting Manifold Regularizationcited by 4
- 2017Enhancing approximation abilities of neural networks by training derivativescited by 3
- 2022Balanced Adversarial Training: Balancing Tradeoffs between Fickleness and Obstinacy in NLP Modelscited by 3
- 2021Pareto Adversarial Robustness: Balancing Spatial Robustness and Sensitivity-based Robustnesscited by 2
- 2023On the Effect of Adversarial Training Against Invariance-based Adversarial Examplescited by 2
- 2022Deep Learning Aided Laplace Based Bayesian Inference for Epidemiological Systemscited by 1
Built on by
Not cited yet.