# EnsembleNet: End-to-End Optimization of Multi-headed Models

arXiv: https://arxiv.org/abs/1905.09979 · HTML: https://mlsanity.com/paper/1905.09979/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2019.

## Built on by (most cited first, 2 of 2)

- [Decentralized Low-Latency Collaborative Inference via Ensembles on the Edge](https://mlsanity.com/paper/2206.03165/) (2022)
- [Probing Contextual Diversity for Dense Out-of-Distribution Detection](https://mlsanity.com/paper/2208.14195/) (2022)

## Abstract (arXiv)

Ensembling is a universally useful approach to boost the performance of machine learning models. However, individual models in an ensemble were traditionally trained independently in separate stages without information access about the overall ensemble. Many co-distillation approaches were proposed in order to treat model ensembling as first-class citizens. In this paper, we reveal a deeper connection between ensembling and distillation, and come up with a simpler yet more effective co-distillation architecture. On large-scale datasets including ImageNet, YouTube-8M, and Kinetics, we demonstrate a general procedure that can convert a single deep neural network to a multi-headed model that has not only a smaller size but also better performance. The model can be optimized end-to-end with our proposed co-distillation loss in a single stage without human intervention.