# Generative Bayesian Hyperparameter Tuning

arXiv: https://arxiv.org/abs/2512.20051 · HTML: https://mlsanity.com/paper/2512.20051/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 23 Dec 2025.

## Builds on (most cited first, 15 of 15)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Bilevel Programming for Hyperparameter Optimization and Meta-Learning](https://mlsanity.com/paper/1806.04910/) (2018)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [A General Framework for Updating Belief Distributions](https://mlsanity.com/paper/1306.6430/) (2013)
- [Merging Two Cultures: Deep and Statistical Learning](https://mlsanity.com/paper/2110.11561/) (2021)
- [Generative Causal Inference](https://mlsanity.com/paper/2306.16096/) (2023)
- [Generative Bayesian Computation for Maximum Expected Utility](https://mlsanity.com/paper/2408.16101/) (2024)
- [Mixtures, envelopes, and hierarchical duality](https://mlsanity.com/paper/1406.0177/) (2014)
- [Generative Multiple-purpose Sampler for Weighted M-estimation](https://mlsanity.com/paper/2006.00767/) (2020)
- [Deep Partial Least Squares for Instrumental Variable Regression](https://mlsanity.com/paper/2207.02612/) (2022)
- [Generative Modeling: A Review](https://mlsanity.com/paper/2501.05458/) (2024)
- [Bayesian Double Descent](https://mlsanity.com/paper/2507.07338/) (2025)
- [Bayesian Global-Local Regularization](https://mlsanity.com/paper/2512.13992/) (2025)

## Abstract (arXiv)

noindent Hyper-parameter selection is a central practical problem in modern machine learning, governing regularization strength, model capacity, and robustness choices. Cross-validation is often computationally prohibitive at scale, while fully Bayesian hyper-parameter learning can be difficult due to the cost of posterior sampling. We develop a generative perspective on hyper-parameter tuning that combines two ideas: (i) optimization-based approximations to Bayesian posteriors via randomized, weighted objectives (weighted Bayesian bootstrap), and (ii) amortization of repeated optimization across many hyper-parameter settings by learning a transport map from hyper-parameters (including random weights) to the corresponding optimizer. This yields a ``generator look-up table'' for estimators, enabling rapid evaluation over grids or continuous ranges of hyper-parameters and supporting both predictive tuning objectives and approximate Bayesian uncertainty quantification. We connect this viewpoint to weighted M-estimation, envelope/auxiliary-variable representations that reduce non-quadratic losses to weighted least squares, and recent generative samplers for weighted M-estimators.