mlsanityData to 7 Oct 2026

Estimating Example Difficulty Using Variance of Gradients

arXiv 2008.11600 ↗First posted 26 Aug 2020cs.CV cs.LGcited by 96

Not on the map. Why?

Abstract (arXiv)

In machine learning, a question of great interest is understanding what examples are challenging for a model to classify. Identifying atypical examples ensures the safe deployment of models, isolates samples that require further human inspection and provides interpretability into model behavior. In this work, we propose Variance of Gradients (VoG) as a valuable and efficient metric to rank data by difficulty and to surface a tractable subset of the most challenging examples for human-in-the-loop auditing. We show that data points with high VoG scores are far more difficult for the model to learn and over-index on corrupted or memorized examples. Further, restricting the evaluation to the test set instances with the lowest VoG improves the model's generalization performance. Finally, we show that VoG is a valuable and efficient ranking for out-of-distribution detection.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 15; 2023: 25; 2024: 20; 2025: 18; 2026: 10.

96 papers.

Sort:

By area:

  1. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  2. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  3. 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
  4. 2022A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Otherscited by 85
  5. 2022Generating High Fidelity Data from Low-density Regions using Diffusion Modelscited by 78
  6. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  7. 2023Discovering and Mitigating Visual Biases through Keyword Explanationcited by 58
  8. 2022Detecting Shortcut Learning for Fair Medical AI using Shortcut Testingcited by 52
  9. 2021Accelerating Deep Learning with Dynamic Data Pruningcited by 43
  10. 2023Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruningcited by 38
  11. 2023Don't Play Favorites: Minority Guidance for Diffusion Modelscited by 37
  12. 2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35
38 more
  1. 2023Data Valuation Without Training of a Modelcited by 35
  2. 2022The Effect of Task Ordering in Continual Learningcited by 30
  3. 2024Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budgetcited by 30
  4. 2022Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamicscited by 26
  5. 2026PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competencecited by 26
  6. 2023Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmarkcited by 23
  7. 2024Weak-to-Strong Generalization Through the Data-Centric Lenscited by 21
  8. 2022How Useful are Gradients for OOD Detection Really?cited by 20
  9. 2022Selective Prediction via Training Dynamicscited by 20
  10. 2021A Tale Of Two Long Tailscited by 19
  11. 2022Deconstructing Distributions: A Pointwise Framework of Learningcited by 19
  12. 2024On the Limitations of Compute Thresholds as a Governance Strategycited by 17
  13. 2023The Memory Perturbation Equation: Understanding Model's Sensitivity to Datacited by 16
  14. 2021Keep CALM and Improve Visual Feature Attributioncited by 15
  15. 2023Learning Sample Difficulty from Pre-trained Models for Reliable Predictioncited by 15
  16. 2023Out-of-Distribution Detection by Leveraging Between-Layer Transformation Smoothnesscited by 15
  17. 2024Cascade-Aware Training of Language Modelscited by 15
  18. 2024Memorization in deep learning: A surveycited by 14
  19. 2024Dataset Difficulty and the Role of Inductive Biascited by 12
  20. 2023Beyond Distribution Shift: Spurious Features Through the Lens of Training Dynamicscited by 11
  21. 2024Efficient Lifelong Model Evaluation in an Era of Rapid Progresscited by 10
  22. 2024Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruningcited by 10
  23. 2022Segmentation Ability Map: Interpret deep features for medical image segmentationcited by 9
  24. 2023Understanding Difficulty-based Sample Weighting with a Universal Difficulty Measurecited by 9
  25. 2023Adaptivity and Modularity for Efficient Generalization Over Task Complexitycited by 7
  26. 2024A Layer Selection Approach to Test Time Adaptationcited by 7
  27. 2024Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformerscited by 7
  28. 2025FUNU: Boosting Machine Unlearning Efficiency by Filtering Unnecessary Unlearningcited by 7
  29. 2023U-Netmer: U-Net meets Transformer for medical image segmentationcited by 6
  30. 2023Data Optimization in Deep Learning: A Surveycited by 6
  31. 2023TRIAGE: Characterizing and auditing training data for improved regressioncited by 6
  32. 2025Trustworthy Machine Learning via Memorization and the Granular Long-Tail: A Survey on Interactions, Tradeoffs, and Beyondcited by 6
  33. 2023Influence Scores at Scale for Efficient Language Data Samplingcited by 5
  34. 2021Beyond Fairness Metrics: Roadblocks and Challenges for Ethical AI in Practicecited by 4
  35. 2022When less is more: Simplifying inputs aids neural network understandingcited by 4
  36. 2022Exploring the Learning Difficulty of Data Theory and Measurecited by 4
  37. 2022On Measuring the Intrinsic Few-Shot Hardness of Datasetscited by 4
  38. 2023The Thousand Faces of Explainable AI Along the Machine Learning Life Cycle: Industrial Reality and Current State of Researchcited by 4

50 most cited of 96 All 96 (JSON)