mlsanityData to 7 Oct 2026

Task Selection for AutoML System Evaluation

arXiv 2208.12754 ↗First posted 26 Aug 2022cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Our goal is to assess if AutoML system changes - i.e., to the search space or hyperparameter optimization - will improve the final model's performance on production tasks. However, we cannot test the changes on production tasks. Instead, we only have access to limited descriptors about tasks that our AutoML system previously executed, like the number of data points or features. We also have a set of development tasks to test changes, ex., sampled from OpenML with no usage constraints. However, the development and production task distributions are different leading us to pursue changes that only improve development and not production. This paper proposes a method to leverage descriptor information about AutoML production tasks to select a filtered subset of the most relevant development tasks. Empirical studies show that our filtering strategy improves the ability to assess AutoML system changes on holdout tasks with different distributions than development.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 33 of its 78 references that are in the map. First posted on arXiv 26 Aug 2022.

Builds on

33 of 78 references.

References · 33

  1. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  2. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  3. 2014OpenML: networked science in machine learningcited by 523
  4. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
  5. 2019AutoML: A Survey of the State-of-the-Artcited by 388
  6. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  7. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  8. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  9. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 277
  10. 2017OpenML Benchmarking Suitescited by 169
  11. 2018Auto-Keras: An Efficient Neural Architecture Search Systemcited by 169
  12. 2021Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020cited by 146
21 more
  1. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 111
  2. 2019Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functionscited by 104
  3. 2019An Open Source AutoML Benchmarkcited by 85
  4. 2021HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenMLcited by 75
  5. 2018Stochastic Hyperparameter Optimization through Hypernetworkscited by 72
  6. 2019Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonflycited by 66
  7. 2016AdaNet: Adaptive Structural Learning of Artificial Neural Networkscited by 54
  8. 2018Rafiki: Machine Learning as an Analytics Service Systemcited by 33
  9. 2018OBOE: Collaborative Filtering for AutoML Model Selectioncited by 31
  10. 2019Automated Machine Learning: State-of-The-Art and Open Challengescited by 29
  11. 2021Meta-Learning to Improve Pre-Trainingcited by 27
  12. 2021AlphaD3M: Machine Learning Pipeline Synthesiscited by 24
  13. 2022Open Source Vizier: Distributed Infrastructure and API for Reliable and Flexible Blackbox Optimizationcited by 17
  14. 2022Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019cited by 16
  15. 2018Benchmarking Automatic Machine Learning Frameworkscited by 15
  16. 2019Understanding Neural Architecture Search Techniquescited by 12
  17. 2018Evolutionary-Neural Hybrid Agents for Architecture Searchcited by 11
  18. 2021Complex Momentum for Optimization in Gamescited by 11
  19. 2021Lyapunov Exponents for Diversity in Differentiable Gamescited by 8
  20. 2020An Adaptive and Near Parameter-free Evolutionary Computation Approach Towards True Automation in AutoMLcited by 4
  21. 2018Neural Architecture Search Over a Graph Search Spacecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 1; 2026: 2.

6 papers.

  1. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  2. 2026Addressable Memory for Video World Modelscited by 11
  3. 2024JacNet: Learning Functions with Structured Jacobianscited by 7
  4. 2024Improving Hyperparameter Optimization with Checkpointed Model Weightscited by 5
  5. 2025Score Distillation Sampling for Audio: Source Separation, Synthesis, and Beyondcited by 5
  6. 2026Motion Attribution for Video Generationcited by 4