# PSEO: Optimizing Post-hoc Stacking Ensemble Through Hyperparameter Tuning

arXiv: https://arxiv.org/abs/2508.05144 · HTML: https://mlsanity.com/paper/2508.05144/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 52 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [Neural Ensemble Search for Uncertainty Estimation and Dataset Shift](https://mlsanity.com/paper/2006.08573/) (2020)
- [OpenBox: A Generalized Black-box Optimization Service](https://mlsanity.com/paper/2106.00421/) (2021)
- [VolcanoML: Speeding up End-to-End AutoML via Scalable Search Space Decomposition](https://mlsanity.com/paper/2107.08861/) (2021)
- [Autostacker: A Compositional Evolutionary Learning System](https://mlsanity.com/paper/1803.00684/) (2018)
- [Bayesian Hyperparameter Optimization for Ensemble Learning](https://mlsanity.com/paper/1605.06394/) (2016)
- [Assembled-OpenML: Creating Efficient Benchmarks for Ensembles in AutoML with OpenML](https://mlsanity.com/paper/2307.00285/) (2023)
- [CMA-ES for Post Hoc Ensembling in AutoML: A Great Success and Salvageable Failure](https://mlsanity.com/paper/2307.00286/) (2023)
- [Q(D)O-ES: Population-based Quality (Diversity) Optimisation for Post Hoc Ensemble Selection in AutoML](https://mlsanity.com/paper/2307.08364/) (2023)
- [DivBO: Diversity-aware CASH for Ensemble Learning](https://mlsanity.com/paper/2302.03255/) (2023)

## Built on by (most cited first, 4 of 4)

- [Uncertainty-Guided Model Selection for Tabular Foundation Models in Biomolecule Efficacy Prediction](https://mlsanity.com/paper/2510.02476/) (2025)
- [Tree-Structured Synergy of Large Language Models and Bayesian Optimization for Efficient CASH](https://mlsanity.com/paper/2601.12355/) (2026)
- [Grokking Is Conditional and Fragile: A Fully-Tractable, Multi-Seed Study at 12K Parameters](https://mlsanity.com/paper/2607.05104/) (2026)
- [CoFEH: LLM-driven Feature Engineering Empowered by Collaborative Bayesian Hyperparameter Optimization](https://mlsanity.com/paper/2602.09851/) (2026)

## Abstract (arXiv)

The Combined Algorithm Selection and Hyperparameter Optimization (CASH) problem is fundamental in Automated Machine Learning (AutoML). Inspired by the success of ensemble learning, recent AutoML systems construct post-hoc ensembles for final predictions rather than relying on the best single model. However, while most CASH methods conduct extensive searches for the optimal single model, they typically employ fixed strategies during the ensemble phase that fail to adapt to specific task characteristics. To tackle this issue, we propose PSEO, a framework for post-hoc stacking ensemble optimization. First, we conduct base model selection through binary quadratic programming, with a trade-off between diversity and performance. Furthermore, we introduce two mechanisms to fully realize the potential of multi-layer stacking. Finally, PSEO builds a hyperparameter space and searches for the optimal post-hoc ensemble strategy within it. Empirical results on 80 public datasets show that sys achieves the best average test rank (2.96) among 16 methods, including post-hoc designs in recent AutoML systems and state-of-the-art ensemble learning methods.