# BrainSegNet: A Novel Framework for Whole-Brain MRI Parcellation Enhanced by Large Models

arXiv: https://arxiv.org/abs/2601.09263 · HTML: https://mlsanity.com/paper/2601.09263/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 14 Jan 2026.

## Builds on (most cited first, 13 of 13)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [Segment Anything in Medical Images](https://mlsanity.com/paper/2304.12306/) (2023)
- [FastSurfer -- A fast and accurate deep learning based neuroimaging pipeline](https://mlsanity.com/paper/1910.03866/) (2019)
- [SegFix: Model-Agnostic Boundary Refinement for Segmentation](https://mlsanity.com/paper/2007.04269/) (2020)
- [Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentation](https://mlsanity.com/paper/2304.12637/) (2023)
- [3D Whole Brain Segmentation using Spatially Localized Atlas Network Tiles](https://mlsanity.com/paper/1903.12152/) (2019)
- [All the attention you need: Global-local, spatial-channel attention for image retrieval](https://mlsanity.com/paper/2107.08000/) (2021)
- [Whole Brain Segmentation with Full Volume Neural Network](https://mlsanity.com/paper/2110.15601/) (2021)
- [Benchmarking the Reproducibility of Brain MRI Segmentation Across Scanners and Time](https://mlsanity.com/paper/2504.15931/) (2025)

## Abstract (arXiv)

Whole-brain parcellation from MRI is a critical yet challenging task due to the complexity of subdividing the brain into numerous small, irregular shaped regions. Traditionally, template-registration methods were used, but recent advances have shifted to deep learning for faster workflows. While large models like the Segment Anything Model (SAM) offer transferable feature representations, they are not tailored for the high precision required in brain parcellation. To address this, we propose BrainSegNet, a novel framework that adapts SAM for accurate whole-brain parcellation into 95 regions. We enhance SAM by integrating U-Net skip connections and specialized modules into its encoder and decoder, enabling fine-grained anatomical precision. Key components include a hybrid encoder combining U-Net skip connections with SAM's transformer blocks, a multi-scale attention decoder with pyramid pooling for varying-sized structures, and a boundary refinement module to sharpen edges. Experimental results on the Human Connectome Project (HCP) dataset demonstrate that BrainSegNet outperforms several state-of-the-art methods, achieving higher accuracy and robustness in complex, multi-label parcellation.