# Deep Label Fusion: A 3D End-to-End Hybrid Multi-Atlas Segmentation and Deep Learning Pipeline

arXiv: https://arxiv.org/abs/2103.10892 · HTML: https://mlsanity.com/paper/2103.10892/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 21 references that are in the map.
- First posted on arXiv 19 Mar 2021.

## Builds on (most cited first, 8 of 8)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations](https://mlsanity.com/paper/1707.03237/) (2017)
- [Multi-Atlas Segmentation of Biomedical Images: A Survey](https://mlsanity.com/paper/1412.3421/) (2014)
- [3D Deeply Supervised Network for Automatic Liver Segmentation from CT Volumes](https://mlsanity.com/paper/1607.00582/) (2016)
- [Neural Multi-Atlas Label Fusion: Application to Cardiac MR Images](https://mlsanity.com/paper/1709.09641/) (2017)
- [Cross-Modality Multi-Atlas Segmentation Using Deep Neural Networks](https://mlsanity.com/paper/2008.08946/) (2020)
- [VoteNet+ : An Improved Deep Learning Label Fusion Method for Multi-atlas Segmentation](https://mlsanity.com/paper/1911.00582/) (2019)

## Abstract (arXiv)

Deep learning (DL) is the state-of-the-art methodology in various medical image segmentation tasks. However, it requires relatively large amounts of manually labeled training data, which may be infeasible to generate in some applications. In addition, DL methods have relatively poor generalizability to out-of-sample data. Multi-atlas segmentation (MAS), on the other hand, has promising performance using limited amounts of training data and good generalizability. A hybrid method that integrates the high accuracy of DL and good generalizability of MAS is highly desired and could play an important role in segmentation problems where manually labeled data is hard to generate. Most of the prior work focuses on improving single components of MAS using DL rather than directly optimizing the final segmentation accuracy via an end-to-end pipeline. Only one study explored this idea in binary segmentation of 2D images, but it remains unknown whether it generalizes well to multi-class 3D segmentation problems. In this study, we propose a 3D end-to-end hybrid pipeline, named deep label fusion (DLF), that takes advantage of the strengths of MAS and DL. Experimental results demonstrate that DLF yields significant improvements over conventional label fusion methods and U-Net, a direct DL approach, in the context of segmenting medial temporal lobe subregions using 3T T1-weighted and T2-weighted MRI. Further, when applied to an unseen similar dataset acquired in 7T, DLF maintains its superior performance, which demonstrates its good generalizability.