# Optimizing Ensemble Weights and Hyperparameters of Machine Learning Models for Regression Problems

arXiv: https://arxiv.org/abs/1908.05287 · HTML: https://mlsanity.com/paper/1908.05287/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Aug 2019.

## Built on by (most cited first, 15 of 15)

- [Improving Multimodal Learning via Imbalanced Learning](https://mlsanity.com/paper/2507.10203/) (2025)
- [Test-Time Alignment via Hypothesis Reweighting](https://mlsanity.com/paper/2412.08812/) (2024)
- [Data-Centric Epidemic Forecasting: A Survey](https://mlsanity.com/paper/2207.09370/) (2022)
- [Domain Generalization via Ensemble Stacking for Face Presentation Attack Detection](https://mlsanity.com/paper/2301.02145/) (2023)
- [Sharded Bayesian Additive Regression Trees](https://mlsanity.com/paper/2306.00361/) (2023)
- [Tackling the muon identification in water Cherenkov detectors problem for the future Southern Wide-field Gamma-ray Observatory by means of Machine Learning](https://mlsanity.com/paper/2101.11924/) (2021)
- [MetaStackVis: Visually-Assisted Performance Evaluation of Metamodels](https://mlsanity.com/paper/2212.03539/) (2022)
- [Practical Knowledge Distillation: Using DNNs to Beat DNNs](https://mlsanity.com/paper/2302.12360/) (2023)
- [VBD-MT Chinese-Vietnamese Translation Systems for VLSP 2022](https://mlsanity.com/paper/2308.07601/) (2023)
- [Beyond Simple Averaging: Improving NLP Ensemble Performance with Topological-Data-Analysis-Based Weighting](https://mlsanity.com/paper/2402.14184/) (2024)
- [Dynamic Order Template Prediction for Generative Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2406.11130/) (2024)
- [Enhancing supply chain security with automated machine learning](https://mlsanity.com/paper/2406.13166/) (2024)
- [Learning a Fast 3D Spectral Approach to Object Segmentation and Tracking over Space and Time](https://mlsanity.com/paper/2212.08058/) (2022)
- [Regularized boosting with an increasing coefficient magnitude stop criterion as meta-learner in hyperparameter optimization stacking ensemble](https://mlsanity.com/paper/2402.01379/) (2024)
- [DRFormer: A Dual-Regularized Bidirectional Transformer for Person Re-identification](https://mlsanity.com/paper/2602.01059/) (2026)

## Abstract (arXiv)

Aggregating multiple learners through an ensemble of models aim to make better predictions by capturing the underlying distribution of the data more accurately. Different ensembling methods, such as bagging, boosting, and stacking/blending, have been studied and adopted extensively in research and practice. While bagging and boosting focus more on reducing variance and bias, respectively, stacking approaches target both by finding the optimal way to combine base learners. In stacking with the weighted average, ensembles are created from weighted averages of multiple base learners. It is known that tuning hyperparameters of each base learner inside the ensemble weight optimization process can produce better performing ensembles. To this end, an optimization-based nested algorithm that considers tuning hyperparameters as well as finding the optimal weights to combine ensembles (Generalized Weighted Ensemble with Internally Tuned Hyperparameters (GEM-ITH)) is designed. Besides, Bayesian search was used to speed-up the optimizing process, and a heuristic was implemented to generate diverse and well-performing base learners. The algorithm is shown to be generalizable to real data sets through analyses with ten publicly available data sets.