# Critical Hyper-Parameters: No Random, No Cry

arXiv: https://arxiv.org/abs/1706.03200 · HTML: https://mlsanity.com/paper/1706.03200/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2017.

## Built on by (most cited first, 8 of 8)

- [Accelerating Neural Network Training: An Analysis of the AlgoPerf Competition](https://mlsanity.com/paper/2502.15015/) (2025)
- [A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes](https://mlsanity.com/paper/2102.06356/) (2021)
- [carps: A Framework for Comparing N Hyperparameter Optimizers on M Benchmarks](https://mlsanity.com/paper/2506.06143/) (2025)
- [How far away are truly hyperparameter-free learning algorithms?](https://mlsanity.com/paper/2505.24005/) (2025)
- [Training neural networks faster with minimal tuning using pre-computed lists of hyperparameters for NAdamW](https://mlsanity.com/paper/2503.03986/) (2025)
- [Surrogate Model Based Hyperparameter Tuning for Deep Learning with SPOT](https://mlsanity.com/paper/2105.14625/) (2021)
- [What do near-optimal learning rate schedules look like?](https://mlsanity.com/paper/2603.10301/) (2026)
- [Constitutional adapters: Inference-time interventions for misalignment and misuse](https://mlsanity.com/paper/2609.36657/) (2026)

## Abstract (arXiv)

The selection of hyper-parameters is critical in Deep Learning. Because of the long training time of complex models and the availability of compute resources in the cloud, "one-shot" optimization schemes - where the sets of hyper-parameters are selected in advance (e.g. on a grid or in a random manner) and the training is executed in parallel - are commonly used. It is known that grid search is sub-optimal, especially when only a few critical parameters matter, and suggest to use random search instead. Yet, random search can be "unlucky" and produce sets of values that leave some part of the domain unexplored. Quasi-random methods, such as Low Discrepancy Sequences (LDS) avoid these issues. We show that such methods have theoretical properties that make them appealing for performing hyperparameter search, and demonstrate that, when applied to the selection of hyperparameters of complex Deep Learning models (such as state-of-the-art LSTM language models and image classification models), they yield suitable hyperparameters values with much fewer runs than random search. We propose a particularly simple LDS method which can be used as a drop-in replacement for grid or random search in any Deep Learning pipeline, both as a fully one-shot hyperparameter search or as an initializer in iterative batch optimization.