mlsanityData to 14 Oct 2026

AMLB: an AutoML Benchmark

arXiv 2207.12560 ↗First posted 25 Jul 2022cs.LG stat.MLcited by 55

Not on the map. Why?

Abstract (arXiv)

Comparing different AutoML frameworks is notoriously challenging and often done incorrectly. We introduce an open and extensible benchmark that follows best practices and avoids common mistakes when comparing AutoML frameworks. We conduct a thorough comparison of 9 well-known AutoML frameworks across 71 classification and 33 regression tasks. The differences between the AutoML frameworks are explored with a multi-faceted analysis, evaluating model accuracy, its trade-offs with inference time, and framework failures. We also use Bradley-Terry trees to discover subsets of tasks where the relative AutoML framework rankings differ. The benchmark comes with an open-source tool that integrates with many AutoML frameworks and automates the empirical evaluation process end-to-end: from framework installation and resource allocation to in-depth evaluation. The benchmark uses public data sets, can be easily extended with other AutoML frameworks and tasks, and has a website with up-to-date results.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Builds on 32 of its 74 references that are in the map. First posted on arXiv 25 Jul 2022.

Builds on

32 of 74 references.

References · 32

  1. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  2. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
  3. 2014OpenML: networked science in machine learningcited by 526
  4. 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 383
  5. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 221
  6. 2017OpenML Benchmarking Suitescited by 178
  7. 2020Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learningcited by 174
  8. 2018Auto-Keras: An Efficient Neural Architecture Search Systemcited by 171
  9. 2017PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparisoncited by 140
  10. 2016COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Settingcited by 138
  11. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 112
  12. 2019FLAML: A Fast and Lightweight AutoML Librarycited by 106
20 more
  1. 2018Tunability: Importance of Hyperparameters of Machine Learning Algorithmscited by 99
  2. 2019An Open Source AutoML Benchmarkcited by 85
  3. 2016Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Sciencecited by 75
  4. 2021HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenMLcited by 75
  5. 2020Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDLcited by 67
  6. 2020NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Searchcited by 64
  7. 2017Hyperparameter Importance Across Datasetscited by 63
  8. 2021LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lassocited by 41
  9. 2021LightAutoML: AutoML Solution for a Large Financial Services Ecosystemcited by 32
  10. 2021Benchmarking Multimodal AutoML for Tabular Data with Text Fieldscited by 26
  11. 2019Towards Automated Machine Learning: Evaluation and Comparison of AutoML Approaches and Toolscited by 25
  12. 2021AlphaD3M: Machine Learning Pipeline Synthesiscited by 24
  13. 2021Multi-objective Asynchronous Successive Halvingcited by 23
  14. 2020Importance of Tuning Hyperparameters of Machine Learning Algorithmscited by 20
  15. 2019Meta-Surrogate Benchmarking for Hyperparameter Optimizationcited by 17
  16. 2018Benchmarking Automatic Machine Learning Frameworkscited by 15
  17. 2020GAMA: a General Automated Machine learning Assistantcited by 15
  18. 2018Automatic Gradient Boostingcited by 6
  19. 2017Toward the automated analysis of complex diseases in genome-wide association studies using genetic programmingcited by 3
  20. 2020Is deep learning necessary for simple classification tasks?cited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 10; 2024: 15; 2025: 13; 2026: 16.

55 papers.

Sort:

By area:

  1. 2023Data-centric Artificial Intelligence: A Surveycited by 203
  2. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 113
  3. 2024DiscoveryBench: Towards Data-Driven Discovery with Large Language Modelscited by 109
  4. 2024AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMLcited by 87
  5. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  6. 2025Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Modelscited by 77
  7. 2025CausalPFN: Amortized Causal Effect Estimation via In-Context Learningcited by 49
  8. 2023AutoGluon-TimeSeries: AutoML for Probabilistic Time Series Forecastingcited by 36
  9. 2023TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applicationscited by 33
  10. 2022Unsupervised Model Selection for Time-series Anomaly Detectioncited by 25
  11. 2024SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learningcited by 24
  12. 2025TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fieldscited by 18
38 more
  1. 2023Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoMLcited by 16
  2. 2024A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Datacited by 15
  3. 2024From Variability to Stability: Advancing RecSys Benchmarking Practicescited by 12
  4. 2025Decoding-based Regressioncited by 12
  5. 2024HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Modelscited by 10
  6. 2024Large Language Models Engineer Too Many Simple Features For Tabular Datacited by 10
  7. 2023CMA-ES for Post Hoc Ensembling in AutoML: A Great Success and Salvageable Failurecited by 8
  8. 2023Q(D)O-ES: Population-based Quality (Diversity) Optimisation for Post Hoc Ensemble Selection in AutoMLcited by 8
  9. 2024Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimizationcited by 7
  10. 2024Bag of Tricks for Multimodal AutoML with Image, Text, and Tabular Datacited by 7
  11. 2023Multi-Objective Optimization of Performance and Interpretability of Tabular Supervised Machine Learning Modelscited by 6
  12. 2024Don't Waste Your Time: Early Stopping Cross-Validationcited by 6
  13. 2024PMLBmini: A Tabular Classification Benchmark Suite for Data-Scarce Applicationscited by 6
  14. 2026ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rulescited by 6
  15. 2026TabClustPFN: A Prior-data Fitted Network for Tabular Data Clusteringcited by 5
  16. 2025Measuring Data Science Automation: A Survey of Evaluation Tools for AI Assistants and Agentscited by 4
  17. 2024Fusing Echocardiography Images and Medical Records for Continuous Patient Stratificationcited by 3
  18. 2025COPA: Comparing the incomparable in multi-objective model evaluationcited by 3
  19. 2026Engineering Regression Without Real-Data Training: Domain Adaptation for Tabular Foundation Models Using Multi-Dataset Embeddingscited by 3
  20. 2024BanditCAT and AutoIRT: Machine Learning Approaches to Computerized Adaptive Testing and Item Calibrationcited by 2
  21. 2026Solve for the Hyperparameter, Skip the Search: Kolmogorov-Optimal Scaling Laws for Spline Regressioncited by 2
  22. 2023SeFNet: Bridging Tabular Datasets with Semantic Feature Netscited by 1
  23. 2025EDCA - An Evolutionary Data-Centric AutoML Framework for Efficient Pipelinescited by 1
  24. 2025AutoML Benchmark with shorter time constraints and early stoppingcited by 1
  25. 2025How Usable is Automated Feature Engineering for Tabular Data?cited by 1
  26. 2026Embedding-Aware Feature Discovery: Bridging Latent Representations and Interpretable Features in Event Sequencescited by 1
  27. 2026CellScientist: From Execution Feedback to Auditable Model-Revision Trajectories for Cellular Perturbation Predictioncited by 1
  28. 2026Winning by Peeking: Unenforced Budgets and Test-Set Selection Inflate Short-Budget AutoML Comparisonscited by 1
  29. 2023STREAMLINE: An Automated Machine Learning Pipeline for Biomedicine Applied to Examine the Utility of Photography-Based Phenotypes for OSA Prediction Across International Sleep Centerscited by 0
  30. 2024Integrating Hyperparameter Search into Model-Free AutoML with Context-Free Grammarscited by 0
  31. 2025VirnyFlow: Optimizing ML Pipelines for Accuracy, Fairness, and Stability at Scalecited by 0
  32. 2025Interpretable by Design: MH-AutoML for Transparent and Efficient Android Malware Detection without Compromising Performancecited by 0
  33. 2025Multi-layer Stack Ensembles for Time Series Forecastingcited by 0
  34. 2025Best Practices For Empirical Meta-Algorithmic Research: Guidelines from the COSEAL Research Networkcited by 0
  35. 2026Integrating Meta-Features with Knowledge Graph Embeddings for Meta-Learningcited by 0
  36. 2026From Code to Prediction: Fine-Tuning LLMs for Neural Network Performance Classification in NNGPTcited by 0
  37. 2026LLMSYS-HPOBench: Hyperparameter Optimization Benchmark Suite for Real-World LLM Systemscited by 0
  38. 2026AGOP-IxG: A Gradient Covariance Filter for Local Feature Attribution on Tabular Data, with a Controlled Benchmarkcited by 0

50 most cited of 55 All 55 (JSON)