# DIVA: Dataset Derivative of a Learning Task

arXiv: https://arxiv.org/abs/2111.09785 · HTML: https://mlsanity.com/paper/2111.09785/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 75 references that are in the map.
- First posted on arXiv 18 Nov 2021.

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020)
- [Training Region-based Object Detectors with Online Hard Example Mining](https://mlsanity.com/paper/1604.03540/) (2016)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [AutoML: A Survey of the State-of-the-Art](https://mlsanity.com/paper/1908.00709/) (2019)
- [Selection via Proxy: Efficient Data Selection for Deep Learning](https://mlsanity.com/paper/1906.11829/) (2019)
- [Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation](https://mlsanity.com/paper/1901.02985/) (2019)
- [What is the Effect of Importance Weighting in Deep Learning?](https://mlsanity.com/paper/1812.03372/) (2018)
- [Fast AutoAugment](https://mlsanity.com/paper/1905.00397/) (2019)
- [Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples](https://mlsanity.com/paper/1704.07433/) (2017)
- [Gradient Harmonized Single-stage Detector](https://mlsanity.com/paper/1811.05181/) (2018)
- [Deep Imbalanced Learning for Face Recognition and Attribute Prediction](https://mlsanity.com/paper/1806.00194/) (2018)
- [Rethinking the Hyperparameters for Fine-tuning](https://mlsanity.com/paper/2002.11770/) (2020)
- [Class Rectification Hard Mining for Imbalanced Deep Learning](https://mlsanity.com/paper/1712.03162/) (2017)
- [DADA: Differentiable Automatic Data Augmentation](https://mlsanity.com/paper/2003.03780/) (2020)
- [Deep Bilevel Learning](https://mlsanity.com/paper/1809.01465/) (2018)
- [A linearized framework and a new benchmark for model selection for fine-tuning](https://mlsanity.com/paper/2102.00084/) (2021)
- [Understanding the role of importance weighting for deep learning](https://mlsanity.com/paper/2103.15209/) (2021)
- [Gradients as Features for Deep Representation Learning](https://mlsanity.com/paper/2004.05529/) (2020)
- [Direct Differentiable Augmentation Search](https://mlsanity.com/paper/2104.04282/) (2021)

## Built on by (most cited first, 5 of 5)

- [Improved Fine-Tuning by Better Leveraging Pre-Training Data](https://mlsanity.com/paper/2111.12292/) (2021)
- [Gradient-based Bi-level Optimization for Deep Learning: A Survey](https://mlsanity.com/paper/2207.11719/) (2022)
- [Experimental Design for Active Transductive Inference in Large Language Models](https://mlsanity.com/paper/2404.08846/) (2024)
- [Annotation-Free Human Sketch Quality Assessment](https://mlsanity.com/paper/2507.20548/) (2025)
- [Position: Unlabeled IS NOT Equal to No Human Supervision in Visual Learning](https://mlsanity.com/paper/2609.03077/) (2026)

## Abstract (arXiv)

We present a method to compute the derivative of a learning task with respect to a dataset. A learning task is a function from a training set to the validation error, which can be represented by a trained deep neural network (DNN). The "dataset derivative" is a linear operator, computed around the trained model, that informs how perturbations of the weight of each training sample affect the validation error, usually computed on a separate validation dataset. Our method, DIVA (Differentiable Validation) hinges on a closed-form differentiable expression of the leave-one-out cross-validation error around a pre-trained DNN. Such expression constitutes the dataset derivative. DIVA could be used for dataset auto-curation, for example removing samples with faulty annotations, augmenting a dataset with additional relevant samples, or rebalancing. More generally, DIVA can be used to optimize the dataset, along with the parameters of the model, as part of the training process without the need for a separate validation dataset, unlike bi-level optimization methods customary in AutoML. To illustrate the flexibility of DIVA, we report experiments on sample auto-curation tasks such as outlier rejection, dataset extension, and automatic aggregation of multi-modal data.