mlsanityData to 7 Oct 2026

A System for Massively Parallel Hyperparameter Tuning

arXiv 1810.05934 ↗First posted 13 Oct 2018cs.LG stat.MLcited by 201

Not on the map. Why?

Abstract (arXiv)

Modern learning models are characterized by large hyperparameter spaces and long training times. These properties, coupled with the rise of parallel computing and the growing demand to productionize machine learning workloads, motivate the need to develop mature hyperparameter optimization functionality in distributed computing settings. We address this challenge by first introducing a simple and robust hyperparameter optimization algorithm called ASHA, which exploits parallelism and aggressive early-stopping to tackle large-scale hyperparameter optimization problems. Our extensive empirical results show that ASHA outperforms existing state-of-the-art hyperparameter optimization methods; scales linearly with the number of workers in distributed settings; and is suitable for massive parallelism, as demonstrated on a task with 500 workers. We then describe several design decisions we encountered, along with our associated solutions, when integrating ASHA in Determined AI's end-to-end production-quality machine learning system that offers hyperparameter tuning as a service.

Key facts. Cited by 201 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 30; 2022: 35; 2023: 37; 2024: 35; 2025: 33; 2026: 31.

201 papers.

Sort:

By area:

  1. 2021Advances in Neural Renderingcited by 365
  2. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  3. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  4. 2021The Benchmark Lotterycited by 133
  5. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 111
  6. 2021How Powerful are Performance Predictors in Neural Architecture Search?cited by 88
  7. 2022Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Trainingcited by 78
  8. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  9. 2021YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimizationcited by 61
  10. 2023Scalable Membership Inference Attacks via Quantile Regressioncited by 61
  11. 2021Model Zoo: A Growing "Brain" That Learns Continuallycited by 53
  12. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 38
38 more
  1. 2022Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overviewcited by 37
  2. 2023Cross-Modal Fine-Tuning: Align then Refinecited by 34
  3. 2021NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Taskscited by 32
  4. 2023PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learningcited by 31
  5. 2023CUDA: Curriculum of Data Augmentation for Long-Tailed Recognitioncited by 30
  6. 2024In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimizationcited by 26
  7. 2022Revisiting Simple Regret: Fast Rates for Returning a Good Armcited by 25
  8. 2023FairTune: Optimizing Parameter Efficient Fine Tuning for Fairness in Medical Image Analysiscited by 23
  9. 2024Specialized Foundation Models Struggle to Beat Supervised Baselinescited by 23
  10. 2022AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuningcited by 22
  11. 2021Automatic Termination for Hyperparameter Optimizationcited by 21
  12. 2021NAS-Bench-x11 and the Power of Learning Curvescited by 20
  13. 2021Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimizationcited by 19
  14. 2022Efficient Architecture Search for Diverse Taskscited by 18
  15. 2022Supervising the Multi-Fidelity Race of Hyperparameter Configurationscited by 15
  16. 2021AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Flycited by 14
  17. 2022Machine Learning for Particle Flow Reconstruction at CMScited by 14
  18. 2024ImageNot: A contrast with ImageNet preserves model rankingscited by 14
  19. 2022Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scalecited by 13
  20. 2022PASHA: Efficient HPO and NAS with Progressive Resource Allocationcited by 13
  21. 2023Event-Based Contrastive Learning for Medical Time Seriescited by 13
  22. 2021An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Modelscited by 12
  23. 2024Efficient Data Subset Selection to Generalize Training Across Models: Transductive and Inductive Networkscited by 12
  24. 2021Tune It or Don't Use It: Benchmarking Data-Efficient Image Classificationcited by 11
  25. 2024Koopman-Assisted Reinforcement Learningcited by 11
  26. 2024HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Modelscited by 10
  27. 2022Open-source FPGA-ML codesign for the MLPerf Tiny Benchmarkcited by 9
  28. 2023MILO: Model-Agnostic Subset Selection Framework for Efficient Model Training and Tuningcited by 9
  29. 2023Machine learning detects terminal singularitiescited by 9
  30. 2024Hyperband-based Bayesian Optimization for Black-box Prompt Selectioncited by 9
  31. 2021Self-attention Presents Low-dimensional Knowledge Graph Embeddings for Link Predictioncited by 8
  32. 2021Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiationcited by 7
  33. 2024Input-to-State Stable Coupled Oscillator Networks for Closed-form Model-based Control in Latent Spacecited by 7
  34. 2021Machine learning classification of non-Markovian noise disturbing quantum dynamicscited by 6
  35. 2021RIANN -- A Robust Neural Network Outperforms Attitude Estimation Filterscited by 6
  36. 2021FairAutoML: Embracing Unfairness Mitigation in AutoMLcited by 6
  37. 2022Asynchronous Decentralized Bayesian Optimization for Large Scale Hyperparameter Optimizationcited by 6
  38. 2022Speeding up NAS with Adaptive Subset Selectioncited by 6

50 most cited of 201 All 201 (JSON)