DREAM: Debugging and Repairing AutoML Pipelines
Not on the map. Why?
Abstract (arXiv)
Deep Learning models have become an integrated component of modern software systems. In response to the challenge of model design, researchers proposed Automated Machine Learning (AutoML) systems, which automatically search for model architecture and hyperparameters for a given task. Like other software systems, existing AutoML systems suffer from bugs. We identify two common and severe bugs in AutoML, performance bug (i.e., searching for the desired model takes an unreasonably long time) and ineffective search bug (i.e., AutoML systems are not able to find an accurate enough model). After analyzing the workflow of AutoML, we observe that existing AutoML systems overlook potential opportunities in search space, search method, and search feedback, which results in performance and ineffective search bugs. Based on our analysis, we design and implement DREAM, an automatic debugging and repairing system for AutoML systems. It monitors the process of AutoML to collect detailed feedback and automatically repairs bugs by expanding search space and leveraging a feedback-driven search strategy. Our evaluation results show that DREAM can effectively and efficiently repair AutoML bugs.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 70 of its 152 references that are in the map. First posted on arXiv 31 Dec 2023.
Builds on
70 of 152 references.
References · 70
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
- 2018DARTS: Differentiable Architecture Searchcited by 1,782
- 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
- 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
- 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
- 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
- 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
- 2017Self-Normalizing Neural Networkscited by 591
38 more
- 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
- 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
- 2019AutoML: A Survey of the State-of-the-Artcited by 388
- 2017Large-Scale Evolution of Image Classifierscited by 348
- 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 307
- 2016Designing Neural Network Architectures using Reinforcement Learningcited by 277
- 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
- 2016Hyperparameter optimization with approximate gradientcited by 231
- 2018Neural Architecture Search with Bayesian Optimisation and Optimal Transportcited by 192
- 2019BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Searchcited by 175
- 2020Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learningcited by 174
- 2018Auto-Keras: An Efficient Neural Architecture Search Systemcited by 169
- 2018Neural Architecture Optimizationcited by 164
- 2019Population Based Augmentation: Efficient Learning of Augmentation Policy Schedulescited by 153
- 2018DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systemscited by 144
- 2016Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasetscited by 131
- 2017Genetic CNNcited by 124
- 2018Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Searchcited by 122
- 2019On NMT Search Errors and Model Errors: Cat Got Your Tongue?cited by 114
- 2018A Survey on Data Collection for Machine Learning: a Big Data -- AI Integration Perspectivecited by 111
- 2019Exploring Randomly Wired Neural Networks for Image Recognitioncited by 103
- 2018Improving Automatic Source Code Summarization via Deep Reinforcement Learningcited by 99
- 2017Data Noising as Smoothing in Neural Network Language Modelscited by 78
- 2018Automated Directed Fairness Testingcited by 77
- 2018TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzingcited by 67
- 2019The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squarescited by 67
- 2018Stable Recurrent Modelscited by 65
- 2018DLFuzz: Differential Fuzzing Testing of Deep Learning Systemscited by 64
- 2017A Genetic Programming Approach to Designing Convolutional Neural Network Architecturescited by 59
- 2020MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulationcited by 56
- 2018Concolic Testing for Deep Neural Networkscited by 51
- 2018DeepBillboard: Systematic Physical-World Testing of Autonomous Driving Systemscited by 50
- 2021NeuronFair: Interpretable White-Box Fairness Testing through Biased Neuron Identificationcited by 42
- 2020Best-First Beam Searchcited by 40
- 2018Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Searchcited by 38
- 2018Collapse of Deep and Narrow Neural Netscited by 37
- 2021DeepLocalize: Fault Localization for Deep Neural Networkscited by 37
- 2019Automatic Testing and Improvement of Machine Translationcited by 35
50 most cited of 70 All 70 (JSON)
Built on by
Citing papers per year: 2025: 1.
1 paper.