# Coupled Ensembles of Neural Networks

arXiv: https://arxiv.org/abs/1709.06053 · HTML: https://mlsanity.com/paper/1709.06053/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2017.

## Built on by (most cited first, 8 of 8)

- [Universal Simultaneous Machine Translation with Mixture-of-Experts Wait-k Policy](https://mlsanity.com/paper/2109.05238/) (2021)
- [Joint Training of Deep Ensembles Fails Due to Learner Collusion](https://mlsanity.com/paper/2301.11323/) (2023)
- [Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learning](https://mlsanity.com/paper/2405.17613/) (2024)
- [Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training](https://mlsanity.com/paper/2107.05736/) (2021)
- [Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy Annotations](https://mlsanity.com/paper/2107.04746/) (2021)
- [Facial Expressions Recognition with Convolutional Neural Networks](https://mlsanity.com/paper/2107.08640/) (2021)
- [On the Effectiveness of Neural Ensembles for Image Classification with Small Datasets](https://mlsanity.com/paper/2111.14493/) (2021)
- [Multi-Narrow Transformation as a Single-Model Ensemble: Boundary Conditions, Mechanisms, and Failure Modes](https://mlsanity.com/paper/2605.11530/) (2026)

## Abstract (arXiv)

We investigate in this paper the architecture of deep convolutional networks. Building on existing state of the art models, we propose a reconfiguration of the model parameters into several parallel branches at the global network level, with each branch being a standalone CNN. We show that this arrangement is an efficient way to significantly reduce the number of parameters without losing performance or to significantly improve the performance with the same level of performance. The use of branches brings an additional form of regularization. In addition to the split into parallel branches, we propose a tighter coupling of these branches by placing the "fuse (averaging) layer" before the Log-Likelihood and SoftMax layers during training. This gives another significant performance improvement, the tighter coupling favouring the learning of better representations, even at the level of the individual branches. We refer to this branched architecture as "coupled ensembles". The approach is very generic and can be applied with almost any DCNN architecture. With coupled ensembles of DenseNet-BC and parameter budget of 25M, we obtain error rates of 2.92%, 15.68% and 1.50% respectively on CIFAR-10, CIFAR-100 and SVHN tasks. For the same budget, DenseNet-BC has error rate of 3.46%, 17.18%, and 1.8% respectively. With ensembles of coupled ensembles, of DenseNet-BC networks, with 50M total parameters, we obtain error rates of 2.72%, 15.13% and 1.42% respectively on these tasks.