mlsanityData to 14 Oct 2026

An Open Source AutoML Benchmark

arXiv 1907.00909 ↗First posted 1 Jul 2019cs.LG stat.MLcited by 85

Not on the map. Why?

Abstract (arXiv)

In recent years, an active field of research has developed around automated machine learning (AutoML). Unfortunately, comparing different AutoML systems is hard and often done incorrectly. We introduce an open, ongoing, and extensible benchmark framework which follows best practices and avoids common mistakes. The framework is open-source, uses public datasets and has a website with up-to-date results. We use the framework to conduct a thorough comparison of 4 AutoML systems across 39 datasets and analyze the results.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). First posted on arXiv 1 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 22; 2022: 16; 2023: 22; 2024: 13; 2025: 4; 2026: 8.

85 papers.

Sort:

By area:

  1. 2021Deep Neural Networks and Tabular Data: A Surveycited by 491
  2. 2021Transformers Can Do Bayesian Inferencecited by 360
  3. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 223
  4. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 199
  5. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  6. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 112
  7. 2021CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmarkcited by 101
  8. 2023Benchmarking Distribution Shift in Tabular Data with TableShiftcited by 66
  9. 2022AMLB: an AutoML Benchmarkcited by 55
  10. 2023Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networkscited by 37
  11. 2023TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applicationscited by 33
  12. 2021LightAutoML: AutoML Solution for a Large Financial Services Ecosystemcited by 32
38 more
  1. 2023A Large-Scale Study of Probabilistic Calibration in Neural Network Regressioncited by 27
  2. 2021Benchmarking Multimodal AutoML for Tabular Data with Text Fieldscited by 26
  3. 2024In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimizationcited by 26
  4. 2021Towards Green Automated Machine Learning: Status Quo and Future Directionscited by 25
  5. 2022MLOps -- Definitions, Tools and Challengescited by 25
  6. 2022Unsupervised Model Selection for Time-series Anomaly Detectioncited by 25
  7. 2024Kolb-Based Experiential Learning for Generalist Agents with Human-Level Kaggle Data Science Performancecited by 25
  8. 2021Bayesian Optimization over Hybrid Spacescited by 23
  9. 2023Scaling Laws for Hyperparameter Optimizationcited by 16
  10. 2025TabPFN-Wide: Continued Pre-Training for Extreme Feature Countscited by 16
  11. 2026The Last Human-Written Paper: Agent-Native Research Artifactscited by 16
  12. 2021Fast and Informative Model Selection using Learning Curve Cross-Validationcited by 15
  13. 2024A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Datacited by 15
  14. 2024AutoM3L: An Automated Multimodal Machine Learning Framework with Large Language Modelscited by 13
  15. 2022Online AutoML: An adaptive AutoML framework for online learningcited by 11
  16. 20244DBInfer: A 4D Benchmarking Toolbox for Graph-Centric Predictive Modeling on Relational DBscited by 11
  17. 2021EXPObench: Benchmarking Surrogate-based Optimisation Algorithms on Expensive Black-box Functionscited by 9
  18. 2023OpenBox: A Python Toolkit for Generalized Black-box Optimizationcited by 9
  19. 2026From Zero to Hero: Advancing Zero-Shot Foundation Models for Tabular Outlier Detectioncited by 9
  20. 2021Naive Automated Machine Learningcited by 8
  21. 2022SubStrat: A Subset-Based Strategy for Faster AutoMLcited by 8
  22. 2021A Scalable AutoML Approach Based on Graph Neural Networkscited by 7
  23. 2022Task Selection for AutoML System Evaluationcited by 7
  24. 2022Fraud Dataset Benchmark and Applicationscited by 7
  25. 2021Incremental Search Space Construction for Machine Learning Pipeline Synthesiscited by 6
  26. 2022FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimizationcited by 6
  27. 2023Interpretable Mesomorphic Networks for Tabular Datacited by 6
  28. 2023AutoXPCR: Automated Multi-Objective Model Selection for Time Series Forecastingcited by 6
  29. 2024Probabilistic Calibration by Design for Neural Network Regressioncited by 6
  30. 2024Don't Waste Your Time: Early Stopping Cross-Validationcited by 6
  31. 2024PMLBmini: A Tabular Classification Benchmark Suite for Data-Scarce Applicationscited by 6
  32. 2022Automated machine learning: AI-driven decision making in business analyticscited by 5
  33. 2023Auto-FP: An Experimental Study of Automated Feature Preprocessing for Tabular Datacited by 5
  34. 2026MacrOData: New Benchmarks of Thousands of Datasets for Tabular Outlier Detectioncited by 5
  35. 2021Meta-Learning for Symbolic Hyperparameter Defaultscited by 4
  36. 2021Amortized Auto-Tuning: Cost-Efficient Bayesian Transfer Optimization for Hyperparameter Recommendationcited by 4
  37. 2022Machine Learning in Heterogeneous Porous Materialscited by 3
  38. 2022AutoMLBench: A Comprehensive Experimental Evaluation of Automated Machine Learning Frameworkscited by 3

50 most cited of 85 All 85 (JSON)