# Estimating Example Difficulty Using Variance of Gradients

arXiv: https://arxiv.org/abs/2008.11600 · HTML: https://mlsanity.com/paper/2008.11600/

## Key facts

- Cited by 97 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Aug 2020.

## Built on by (most cited first, 30 of 97)

- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Deep Learning Through the Lens of Example Difficulty](https://mlsanity.com/paper/2106.09647/) (2021)
- [HIVE: Evaluating the Human Interpretability of Visual Explanations](https://mlsanity.com/paper/2112.03184/) (2021)
- [A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others](https://mlsanity.com/paper/2212.04825/) (2022)
- [Generating High Fidelity Data from Low-density Regions using Diffusion Models](https://mlsanity.com/paper/2203.17260/) (2022)
- [Efficient Methods for Natural Language Processing: A Survey](https://mlsanity.com/paper/2209.00099/) (2022)
- [Discovering and Mitigating Visual Biases through Keyword Explanation](https://mlsanity.com/paper/2301.11104/) (2023)
- [Detecting Shortcut Learning for Fair Medical AI using Shortcut Testing](https://mlsanity.com/paper/2207.10384/) (2022)
- [Accelerating Deep Learning with Dynamic Data Pruning](https://mlsanity.com/paper/2111.12621/) (2021)
- [Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning](https://mlsanity.com/paper/2311.13613/) (2023)
- [Don't Play Favorites: Minority Guidance for Diffusion Models](https://mlsanity.com/paper/2301.12334/) (2023)
- [Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)](https://mlsanity.com/paper/2110.05922/) (2021)
- [Data Valuation Without Training of a Model](https://mlsanity.com/paper/2301.00930/) (2023)
- [The Effect of Task Ordering in Continual Learning](https://mlsanity.com/paper/2205.13323/) (2022)
- [Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget](https://mlsanity.com/paper/2407.15811/) (2024)
- [PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence](https://mlsanity.com/paper/2603.11178/) (2026)
- [Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics](https://mlsanity.com/paper/2209.10015/) (2022)
- [Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmark](https://mlsanity.com/paper/2310.16981/) (2023)
- [Weak-to-Strong Generalization Through the Data-Centric Lens](https://mlsanity.com/paper/2412.03881/) (2024)
- [How Useful are Gradients for OOD Detection Really?](https://mlsanity.com/paper/2205.10439/) (2022)
- [Selective Prediction via Training Dynamics](https://mlsanity.com/paper/2205.13532/) (2022)
- [A Tale Of Two Long Tails](https://mlsanity.com/paper/2107.13098/) (2021)
- [Deconstructing Distributions: A Pointwise Framework of Learning](https://mlsanity.com/paper/2202.09931/) (2022)
- [On the Limitations of Compute Thresholds as a Governance Strategy](https://mlsanity.com/paper/2407.05694/) (2024)
- [The Memory Perturbation Equation: Understanding Model's Sensitivity to Data](https://mlsanity.com/paper/2310.19273/) (2023)
- [Keep CALM and Improve Visual Feature Attribution](https://mlsanity.com/paper/2106.07861/) (2021)
- [Learning Sample Difficulty from Pre-trained Models for Reliable Prediction](https://mlsanity.com/paper/2304.10127/) (2023)
- [Out-of-Distribution Detection by Leveraging Between-Layer Transformation Smoothness](https://mlsanity.com/paper/2310.02832/) (2023)
- [Cascade-Aware Training of Language Models](https://mlsanity.com/paper/2406.00060/) (2024)
- [Memorization in deep learning: A survey](https://mlsanity.com/paper/2406.03880/) (2024)

## Abstract (arXiv)

In machine learning, a question of great interest is understanding what examples are challenging for a model to classify. Identifying atypical examples ensures the safe deployment of models, isolates samples that require further human inspection and provides interpretability into model behavior. In this work, we propose Variance of Gradients (VoG) as a valuable and efficient metric to rank data by difficulty and to surface a tractable subset of the most challenging examples for human-in-the-loop auditing. We show that data points with high VoG scores are far more difficult for the model to learn and over-index on corrupted or memorized examples. Further, restricting the evaluation to the test set instances with the lowest VoG improves the model's generalization performance. Finally, we show that VoG is a valuable and efficient ranking for out-of-distribution detection.