# HyperSched: Dynamic Resource Reallocation for Model Development on a Deadline

arXiv: https://arxiv.org/abs/2001.02338 · HTML: https://mlsanity.com/paper/2001.02338/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jan 2020.

## Built on by (most cited first, 6 of 6)

- [Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision](https://mlsanity.com/paper/2205.11913/) (2022)
- [Scavenger: A Cloud Service for Optimizing Cost and Performance of ML Training](https://mlsanity.com/paper/2303.06659/) (2023)
- [BFTrainer: Low-Cost Training of Neural Networks on Unfillable Supercomputer Nodes](https://mlsanity.com/paper/2106.12091/) (2021)
- [PAC Best Arm Identification Under a Deadline](https://mlsanity.com/paper/2106.03221/) (2021)
- [Towards providing reliable job completion time predictions using PCS](https://mlsanity.com/paper/2401.10354/) (2024)
- [Resource-Adaptive Successive Doubling for Hyperparameter Optimization with Large Datasets on High-Performance Computing Systems](https://mlsanity.com/paper/2412.02729/) (2024)

## Abstract (arXiv)

Prior research in resource scheduling for machine learning training workloads has largely focused on minimizing job completion times. Commonly, these model training workloads collectively search over a large number of parameter values that control the learning process in a hyperparameter search. It is preferable to identify and maximally provision the best-performing hyperparameter configuration (trial) to achieve the highest accuracy result as soon as possible. To optimally trade-off evaluating multiple configurations and training the most promising ones by a fixed deadline, we design and build HyperSched -- a dynamic application-level resource scheduler to track, identify, and preferentially allocate resources to the best performing trials to maximize accuracy by the deadline. HyperSched leverages three properties of a hyperparameter search workload over-looked in prior work - trial disposability, progressively identifiable rankings among different configurations, and space-time constraints - to outperform standard hyperparameter search algorithms across a variety of benchmarks.