# AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuning

arXiv: https://arxiv.org/abs/2203.08212 · HTML: https://mlsanity.com/paper/2203.08212/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 54 references that are in the map.
- First posted on arXiv 15 Mar 2022.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [Scalable Bayesian Optimization Using Deep Neural Networks](https://mlsanity.com/paper/1502.05700/) (2015)
- [GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training](https://mlsanity.com/paper/2103.00123/) (2021)
- [A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay](https://mlsanity.com/paper/1803.09820/) (2018)
- [GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning](https://mlsanity.com/paper/2012.10630/) (2020)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [A System for Massively Parallel Hyperparameter Tuning](https://mlsanity.com/paper/1810.05934/) (2018)
- [Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets](https://mlsanity.com/paper/1605.07079/) (2016)
- [Hyper-Parameter Optimization: A Review of Algorithms and Applications](https://mlsanity.com/paper/2003.05689/) (2020)
- [Freeze-Thaw Bayesian Optimization](https://mlsanity.com/paper/1406.3896/) (2014)
- [RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning](https://mlsanity.com/paper/2106.07760/) (2021)
- [Bayesian Coreset Construction via Greedy Iterative Geodesic Ascent](https://mlsanity.com/paper/1802.01737/) (2018)
- [Learning From Less Data: A Unified Data Subset Selection and Active Learning Framework for Computer Vision](https://mlsanity.com/paper/1901.01151/) (2019)
- [Automated Machine Learning on Big Data using Stochastic Algorithm Tuning](https://mlsanity.com/paper/1407.7969/) (2014)

## Built on by (most cited first, 22 of 22)

- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [Practical Differentially Private Hyperparameter Tuning with Subsampling](https://mlsanity.com/paper/2301.11989/) (2023)
- [SMART: Submodular Data Mixture Strategy for Instruction Tuning](https://mlsanity.com/paper/2403.08370/) (2024)
- [Efficient Data Subset Selection to Generalize Training Across Models: Transductive and Inductive Networks](https://mlsanity.com/paper/2409.12255/) (2024)
- [MILO: Model-Agnostic Subset Selection Framework for Efficient Model Training and Tuning](https://mlsanity.com/paper/2301.13287/) (2023)
- [SCoRe: Submodular Combinatorial Representation Learning](https://mlsanity.com/paper/2310.00165/) (2023)
- [Influential Language Data Selection via Gradient Trajectory Pursuit](https://mlsanity.com/paper/2410.16710/) (2024)
- [Speeding up NAS with Adaptive Subset Selection](https://mlsanity.com/paper/2211.01454/) (2022)
- [A Survey of Dataset Refinement for Problems in Computer Vision Datasets](https://mlsanity.com/paper/2210.11717/) (2022)
- [Contributing Dimension Structure of Deep Feature for Coreset Selection](https://mlsanity.com/paper/2401.16193/) (2024)
- [Looking Beyond the Known: Towards a Data Discovery Guided Open-World Object Detection](https://mlsanity.com/paper/2510.00303/) (2025)
- [ASP: Automatic Selection of Proxy dataset for efficient AutoML](https://mlsanity.com/paper/2310.11478/) (2023)
- [Data Pruning via Separability, Integrity, and Model Uncertainty-Aware Importance Sampling](https://mlsanity.com/paper/2409.13915/) (2024)
- [Two-step hyperparameter optimization method: Accelerating hyperparameter search by using a fraction of a training dataset](https://mlsanity.com/paper/2302.03845/) (2023)
- [STONE: A Submodular Optimization Framework for Active 3D Object Detection](https://mlsanity.com/paper/2410.03918/) (2024)
- [MelissaDL x Breed: Towards Data-Efficient On-line Supervised Training of Multi-parametric Surrogates with Active Learning](https://mlsanity.com/paper/2410.05860/) (2024)
- [InSQuAD: In-Context Learning for Efficient Retrieval via Submodular Mutual Information to Enforce Quality and Diversity](https://mlsanity.com/paper/2508.21003/) (2025)
- [SHaSaM: Submodular Hard Sample Mining for Fair Facial Attribute Recognition](https://mlsanity.com/paper/2602.05162/) (2026)
- [SMA: Submodular Modality Aligner For Data Efficient Multimodal Learning](https://mlsanity.com/paper/2605.12872/) (2026)
- [Comparative study of subset selection methods for rapid prototyping of 3D object detection algorithms](https://mlsanity.com/paper/2306.17551/) (2023)
- [Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection](https://mlsanity.com/paper/2506.14473/) (2025)
- [Position: Stop Preaching and Start Practising Data Frugality for Responsible Development of AI](https://mlsanity.com/paper/2602.19789/) (2026)

## Abstract (arXiv)

Deep neural networks have seen great success in recent years; however, training a deep model is often challenging as its performance heavily depends on the hyper-parameters used. In addition, finding the optimal hyper-parameter configuration, even with state-of-the-art (SOTA) hyper-parameter optimization (HPO) algorithms, can be time-consuming, requiring multiple training runs over the entire dataset for different possible sets of hyper-parameters. Our central insight is that using an informative subset of the dataset for model training runs involved in hyper-parameter optimization, allows us to find the optimal hyper-parameter configuration significantly faster. In this work, we propose AUTOMATA, a gradient-based subset selection framework for hyper-parameter tuning. We empirically evaluate the effectiveness of AUTOMATA in hyper-parameter tuning through several experiments on real-world datasets in the text, vision, and tabular domains. Our experiments show that using gradient-based data subsets for hyper-parameter tuning achieves significantly faster turnaround times and speedups of 3×-30× while achieving comparable performance to the hyper-parameters found using the entire dataset.