# Differentiable Model Selection for Ensemble Learning

arXiv: https://arxiv.org/abs/2211.00251 · HTML: https://mlsanity.com/paper/2211.00251/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 33 references that are in the map.
- First posted on arXiv 1 Nov 2022.

## Builds on (most cited first, 10 of 10)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Age Progression/Regression by Conditional Adversarial Autoencoder](https://mlsanity.com/paper/1702.08423/) (2017)
- [Melding the Data-Decisions Pipeline: Decision-Focused Learning for Combinatorial Optimization](https://mlsanity.com/paper/1809.05504/) (2018)
- [End-to-End Constrained Optimization Learning: A Survey](https://mlsanity.com/paper/2103.16378/) (2021)
- [The Relative Performance of Ensemble Methods with Deep Convolutional Neural Networks for Image Classification](https://mlsanity.com/paper/1704.01664/) (2017)
- [DrMAD: Distilling Reverse-Mode Automatic Differentiation for Optimizing Hyperparameters of Deep Neural Networks](https://mlsanity.com/paper/1601.00917/) (2016)
- [Differentiable Feature Selection, a Reparameterization Approach](https://mlsanity.com/paper/2107.10030/) (2021)

## Built on by (most cited first, 5 of 5)

- [Backpropagation of Unrolled Solvers with Folded Optimization](https://mlsanity.com/paper/2301.12047/) (2023)
- [Towards Robust Multi-Modal Reasoning via Model Selection](https://mlsanity.com/paper/2310.08446/) (2023)
- [Ranked from Within: Ranking Large Multimodal Models Without Labels](https://mlsanity.com/paper/2412.06461/) (2024)
- [Green AI in Action: Strategic Model Selection for Ensembles in Production](https://mlsanity.com/paper/2405.17451/) (2024)
- [The More the Merrier? Navigating Accuracy vs. Energy Efficiency Design Trade-Offs in Ensemble Learning Systems](https://mlsanity.com/paper/2407.02914/) (2024)

## Abstract (arXiv)

Model selection is a strategy aimed at creating accurate and robust models. A key challenge in designing these algorithms is identifying the optimal model for classifying any particular input sample. This paper addresses this challenge and proposes a novel framework for differentiable model selection integrating machine learning and combinatorial optimization. The framework is tailored for ensemble learning, a strategy that combines the outputs of individually pre-trained models, and learns to select appropriate ensemble members for a particular input sample by transforming the ensemble learning task into a differentiable selection program trained end-to-end within the ensemble learning model. Tested on various tasks, the proposed framework demonstrates its versatility and effectiveness, outperforming conventional and advanced consensus rules across a variety of settings and learning tasks.