# Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization

arXiv: https://arxiv.org/abs/1905.04970 · HTML: https://mlsanity.com/paper/1905.04970/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 May 2019.

## Built on by (most cited first, 30 of 58)

- [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021)
- [Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance](https://mlsanity.com/paper/2304.11127/) (2023)
- [HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO](https://mlsanity.com/paper/2109.06716/) (2021)
- [Representation Learning for Tabular Data: A Comprehensive Survey](https://mlsanity.com/paper/2504.16109/) (2025)
- [TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applications](https://mlsanity.com/paper/2311.02971/) (2023)
- [BORE: Bayesian Optimization by Density-Ratio Estimation](https://mlsanity.com/paper/2102.09009/) (2021)
- [NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search](https://mlsanity.com/paper/2206.09166/) (2022)
- [Automatic Termination for Hyperparameter Optimization](https://mlsanity.com/paper/2104.08166/) (2021)
- [A General Recipe for Likelihood-free Bayesian Optimization](https://mlsanity.com/paper/2206.13035/) (2022)
- [Hyperparameter Transfer Learning with Adaptive Complexity](https://mlsanity.com/paper/2102.12810/) (2021)
- [A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data](https://mlsanity.com/paper/2407.02112/) (2024)
- [OptunaHub: A Platform for Black-Box Optimization](https://mlsanity.com/paper/2510.02798/) (2025)
- [Automated Deep Learning: Neural Architecture Search Is Not the End](https://mlsanity.com/paper/2112.09245/) (2021)
- [A multi-objective perspective on jointly tuning hardware and hyperparameters](https://mlsanity.com/paper/2106.05680/) (2021)
- [EC-NAS: Energy Consumption Aware Tabular Benchmarks for Neural Architecture Search](https://mlsanity.com/paper/2210.06015/) (2022)
- [Optimizing Hyperparameters with Conformal Quantile Regression](https://mlsanity.com/paper/2305.03623/) (2023)
- [NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters](https://mlsanity.com/paper/2110.10165/) (2021)
- [c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimization](https://mlsanity.com/paper/2211.14411/) (2022)
- [Assembled-OpenML: Creating Efficient Benchmarks for Ensembles in AutoML with OpenML](https://mlsanity.com/paper/2307.00285/) (2023)
- [JUMBO: Scalable Multi-task Bayesian Optimization using Offline Data](https://mlsanity.com/paper/2106.00942/) (2021)
- [Speeding Up Multi-Objective Hyperparameter Optimization by Task Similarity-Based Meta-Learning for the Tree-Structured Parzen Estimator](https://mlsanity.com/paper/2212.06751/) (2022)
- [Is One Epoch All You Need For Multi-Fidelity Hyperparameter Optimization?](https://mlsanity.com/paper/2307.15422/) (2023)
- [Improving LLM-based Global Optimization with Search Space Partitioning](https://mlsanity.com/paper/2505.21372/) (2025)
- [A resource-efficient method for repeated HPO and NAS problems](https://mlsanity.com/paper/2103.16111/) (2021)
- [The Unreasonable Effectiveness Of Early Discarding After One Epoch In Neural Network Hyperparameter Optimization](https://mlsanity.com/paper/2404.04111/) (2024)
- [Overtuning in Hyperparameter Optimization](https://mlsanity.com/paper/2506.19540/) (2025)
- [DAAS: Differentiable Architecture and Augmentation Policy Search](https://mlsanity.com/paper/2109.15273/) (2021)
- [Pseudo-Bayesian Optimization](https://mlsanity.com/paper/2310.09766/) (2023)
- [Principled Architecture-aware Scaling of Hyperparameters](https://mlsanity.com/paper/2402.17440/) (2024)
- [Hyperparameter Optimization via Interacting with Probabilistic Circuits](https://mlsanity.com/paper/2505.17804/) (2025)

## Abstract (arXiv)

Due to the high computational demands executing a rigorous comparison between hyperparameter optimization (HPO) methods is often cumbersome. The goal of this paper is to facilitate a better empirical evaluation of HPO methods by providing benchmarks that are cheap to evaluate, but still represent realistic use cases. We believe these benchmarks provide an easy and efficient way to conduct reproducible experiments for neural hyperparameter search. Our benchmarks consist of a large grid of configurations of a feed forward neural network on four different regression datasets including architectural hyperparameters and hyperparameters concerning the training pipeline. Based on this data, we performed an in-depth analysis to gain a better understanding of the properties of the optimization problem, as well as of the importance of different types of hyperparameters. Second, we exhaustively compared various different state-of-the-art methods from the hyperparameter optimization literature on these benchmarks in terms of performance and robustness.