# AssemblyNet: A large ensemble of CNNs for 3D Whole Brain MRI Segmentation

arXiv: https://arxiv.org/abs/1911.09098 · HTML: https://mlsanity.com/paper/1911.09098/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2019.

## Built on by (most cited first, 21 of 21)

- [FastSurferVINN: Building Resolution-Independence into Deep Learning Segmentation Methods -- A Solution for HighRes Brain MRI](https://mlsanity.com/paper/2112.09654/) (2021)
- [A Study of Demographic Bias in CNN-based Brain MR Segmentation](https://mlsanity.com/paper/2208.06613/) (2022)
- [Interpretable differential diagnosis for Alzheimer's disease and Frontotemporal dementia](https://mlsanity.com/paper/2206.07417/) (2022)
- [Whole Brain Segmentation with Full Volume Neural Network](https://mlsanity.com/paper/2110.15601/) (2021)
- [Towards better Interpretable and Generalizable AD detection using Collective Artificial Intelligence](https://mlsanity.com/paper/2206.03247/) (2022)
- [Fighting the scanner effect in brain MRI segmentation with a progressive level-of-detail network trained on multi-site data](https://mlsanity.com/paper/2211.02400/) (2022)
- [Deep grading for MRI-based differential diagnosis of Alzheimer's disease and Frontotemporal dementia](https://mlsanity.com/paper/2211.14096/) (2022)
- [DeepCERES: A Deep learning method for cerebellar lobule segmentation using ultra-high resolution multimodal MRI](https://mlsanity.com/paper/2401.12074/) (2024)
- [Brain Structure Ages -- A new biomarker for multi-disease classification](https://mlsanity.com/paper/2304.06591/) (2023)
- [SIAM: Head and Brain MRI Segmentation from Few High-Quality Templates via Synthetic Training](https://mlsanity.com/paper/2605.02737/) (2026)
- [Deep morphological recognition of kidney stones using intra-operative endoscopic digital videos](https://mlsanity.com/paper/2205.06093/) (2022)
- [Joint Reconstruction and Parcellation of Cortical Surfaces](https://mlsanity.com/paper/2210.01772/) (2022)
- [VINNA for Neonates -- Orientation Independence through Latent Augmentations](https://mlsanity.com/paper/2311.17546/) (2023)
- [Deep Grading based on Collective Artificial Intelligence for AD Diagnosis and Prognosis](https://mlsanity.com/paper/2211.15192/) (2022)
- [3D Transformer based on deformable patch location for differential diagnosis between Alzheimer's disease and Frontotemporal dementia](https://mlsanity.com/paper/2309.03183/) (2023)
- [DeepThalamus: A novel deep learning method for automatic segmentation of brain thalamic nuclei from multimodal ultra-high resolution MRI](https://mlsanity.com/paper/2401.07751/) (2024)
- [Ultra-high resolution multimodal MRI densely labelled holistic structural brain atlas](https://mlsanity.com/paper/2501.16879/) (2025)
- [Lifespan tree of brain anatomy: diagnostic values for motor and cognitive neurodegenerative diseases](https://mlsanity.com/paper/2502.09682/) (2025)
- [FLAIRBrainSeg: Fine-grained brain segmentation using FLAIR MRI only](https://mlsanity.com/paper/2504.03376/) (2025)
- [petBrain: A New Pipeline for Amyloid, Tau Tangles and Neurodegeneration Quantification Using PET and MRI](https://mlsanity.com/paper/2506.03217/) (2025)
- [CAHAL: Clinically Applicable resolution enHAncement for Low-resolution MRI scans](https://mlsanity.com/paper/2604.18781/) (2026)

## Abstract (arXiv)

Whole brain segmentation using deep learning (DL) is a very challenging task since the number of anatomical labels is very high compared to the number of available training images. To address this problem, previous DL methods proposed to use a single convolution neural network (CNN) or few independent CNNs. In this paper, we present a novel ensemble method based on a large number of CNNs processing different overlapping brain areas. Inspired by parliamentary decision-making systems, we propose a framework called AssemblyNet, made of two "assemblies" of U-Nets. Such a parliamentary system is capable of dealing with complex decisions, unseen problem and reaching a consensus quickly. AssemblyNet introduces sharing of knowledge among neighboring U-Nets, an "amendment" procedure made by the second assembly at higher-resolution to refine the decision taken by the first one, and a final decision obtained by majority voting. During our validation, AssemblyNet showed competitive performance compared to state-of-the-art methods such as U-Net, Joint label fusion and SLANT. Moreover, we investigated the scan-rescan consistency and the robustness to disease effects of our method. These experiences demonstrated the reliability of AssemblyNet. Finally, we showed the interest of using semi-supervised learning to improve the performance of our method.