# Identifying Untrustworthy Predictions in Neural Networks by Geometric Gradient Analysis

arXiv: https://arxiv.org/abs/2102.12196 · HTML: https://mlsanity.com/paper/2102.12196/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 24 Feb 2021.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [On Adaptive Attacks to Adversarial Example Defenses](https://mlsanity.com/paper/2002.08347/) (2020)
- [Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods](https://mlsanity.com/paper/1705.07263/) (2017)
- [Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples](https://mlsanity.com/paper/2010.03593/) (2020)
- [Explanations can be manipulated and geometry is to blame](https://mlsanity.com/paper/1906.07983/) (2019)
- [To Trust Or Not To Trust A Classifier](https://mlsanity.com/paper/1805.11783/) (2018)
- [Addressing Failure Prediction by Learning Model Confidence](https://mlsanity.com/paper/1910.04851/) (2019)
- [On the (Statistical) Detection of Adversarial Examples](https://mlsanity.com/paper/1702.06280/) (2017)
- [On the Connection Between Adversarial Robustness and Saliency Map Interpretability](https://mlsanity.com/paper/1905.04172/) (2019)
- [Robust Out-of-distribution Detection for Neural Networks](https://mlsanity.com/paper/2003.09711/) (2020)
- [Adversarial Attacks on Deep Neural Networks for Time Series Classification](https://mlsanity.com/paper/1903.07054/) (2019)
- [Accurate, reliable and fast robustness evaluation](https://mlsanity.com/paper/1907.01003/) (2019)
- [With Friends Like These, Who Needs Adversaries?](https://mlsanity.com/paper/1807.04200/) (2018)
- [Saliency Methods for Explaining Adversarial Attacks](https://mlsanity.com/paper/1908.08413/) (2019)
- [Detection Defense Against Adversarial Attacks with Saliency Map](https://mlsanity.com/paper/2009.02738/) (2020)
- [Classification regions of deep neural networks](https://mlsanity.com/paper/1705.09552/) (2017)

## Built on by (most cited first, 13 of 13)

- [Efficient Adversarial Training in LLMs with Continuous Attacks](https://mlsanity.com/paper/2405.15589/) (2024)
- [Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space](https://mlsanity.com/paper/2402.09063/) (2024)
- [Exploring Misclassifications of Robust Neural Networks to Enhance Adversarial Attacks](https://mlsanity.com/paper/2105.10304/) (2021)
- [Improving Robustness against Real-World and Worst-Case Distribution Shifts through Decision Region Quantification](https://mlsanity.com/paper/2205.09619/) (2022)
- [Unsupervised Layer-wise Score Aggregation for Textual OOD Detection](https://mlsanity.com/paper/2302.09852/) (2023)
- [Behind the Machine's Gaze: Neural Networks with Biologically-inspired Constraints Exhibit Human-like Visual Attention](https://mlsanity.com/paper/2204.09093/) (2022)
- [A Unified Approach Towards Active Learning and Out-of-Distribution Detection](https://mlsanity.com/paper/2405.11337/) (2024)
- [Efficient Time Series Processing for Transformers and State-Space Models through Token Merging](https://mlsanity.com/paper/2405.17951/) (2024)
- [Joint Out-of-Distribution Filtering and Data Discovery Active Learning](https://mlsanity.com/paper/2503.02491/) (2025)
- [Diffusion LLMs are Natural Adversaries for any LLM](https://mlsanity.com/paper/2511.00203/) (2025)
- [Byte Pair Encoding for Efficient Time Series Forecasting](https://mlsanity.com/paper/2505.14411/) (2025)
- [It begins with a boundary: A geometric view on probabilistically robust learning](https://mlsanity.com/paper/2305.18779/) (2023)
- [Effective Data Pruning through Score Extrapolation](https://mlsanity.com/paper/2506.09010/) (2025)

## Abstract (arXiv)

The susceptibility of deep neural networks to untrustworthy predictions, including out-of-distribution (OOD) data and adversarial examples, still prevent their widespread use in safety-critical applications. Most existing methods either require a re-training of a given model to achieve robust identification of adversarial attacks or are limited to out-of-distribution sample detection only. In this work, we propose a geometric gradient analysis (GGA) to improve the identification of untrustworthy predictions without retraining of a given model. GGA analyzes the geometry of the loss landscape of neural networks based on the saliency maps of their respective input. To motivate the proposed approach, we provide theoretical connections between gradients' geometrical properties and local minima of the loss function. Furthermore, we demonstrate that the proposed method outperforms prior approaches in detecting OOD data and adversarial attacks, including state-of-the-art and adaptive attacks.