# Model Composition: Can Multiple Neural Networks Be Combined into a Single Network Using Only Unlabeled Data?

arXiv: https://arxiv.org/abs/2110.10369 · HTML: https://mlsanity.com/paper/2110.10369/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 20 Oct 2021.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Soft-NMS -- Improving Object Detection With One Line of Code](https://mlsanity.com/paper/1704.04503/) (2017)
- [Variational Information Distillation for Knowledge Transfer](https://mlsanity.com/paper/1904.05835/) (2019)
- [WIDER FACE: A Face Detection Benchmark](https://mlsanity.com/paper/1511.06523/) (2015)
- [Branched Multi-Task Networks: Deciding What Layers To Share](https://mlsanity.com/paper/1904.02920/) (2019)
- [Faster ILOD: Incremental Learning for Object Detectors based on Faster RCNN](https://mlsanity.com/paper/2003.03901/) (2020)
- [ESL: Entropy-guided Self-supervised Learning for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/2006.08658/) (2020)
- [Weighted boxes fusion: Ensembling boxes from different object detection models](https://mlsanity.com/paper/1910.13302/) (2019)
- [Adaptive Feeding: Achieving Fast and Accurate Detections by Adaptively Combining Object Detectors](https://mlsanity.com/paper/1707.06399/) (2017)
- [Unifying and Merging Well-trained Deep Neural Networks for Inference Stage](https://mlsanity.com/paper/1805.04980/) (2018)
- [Deep Bayesian Active Semi-Supervised Learning](https://mlsanity.com/paper/1803.01216/) (2018)
- [OMNIA Faster R-CNN: Detection in the wild through dataset merging and soft distillation](https://mlsanity.com/paper/1812.02611/) (2018)

## Built on by (most cited first, 2 of 2)

- [Spatial Cross-Attention Improves Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2206.05028/) (2022)
- [On the use of neural networks for the structural characterization of polymeric porous materials](https://mlsanity.com/paper/2502.07076/) (2025)

## Abstract (arXiv)

The diversity of deep learning applications, datasets, and neural network architectures necessitates a careful selection of the architecture and data that match best to a target application. As an attempt to mitigate this dilemma, this paper investigates the idea of combining multiple trained neural networks using unlabeled data. In addition, combining multiple models into one can speed up the inference, result in stronger, more capable models, and allows us to select efficient device-friendly target network architectures. To this end, the proposed method makes use of generation, filtering, and aggregation of reliable pseudo-labels collected from unlabeled data. Our method supports using an arbitrary number of input models with arbitrary architectures and categories. Extensive performance evaluations demonstrated that our method is very effective. For example, for the task of object detection and without using any ground-truth labels, an EfficientDet-D0 trained on Pascal-VOC and an EfficientDet-D1 trained on COCO, can be combined to a RetinaNet-ResNet50 model, with a similar mAP as the supervised training. If fine-tuned in a semi-supervised setting, the combined model achieves +18.6%, +12.6%, and +8.1% mAP improvements over supervised training with 1%, 5%, and 10% of labels.