# BrainSegFounder: Towards 3D Foundation Models for Neuroimage Segmentation

arXiv: https://arxiv.org/abs/2406.10395 · HTML: https://mlsanity.com/paper/2406.10395/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 45 references that are in the map.
- First posted on arXiv 14 Jun 2024.

## Builds on (most cited first, 16 of 16)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [3D MRI brain tumor segmentation using autoencoder regularization](https://mlsanity.com/paper/1810.11654/) (2018)
- [TransBTS: Multimodal Brain Tumor Segmentation Using Transformer](https://mlsanity.com/paper/2103.04430/) (2021)
- [Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation](https://mlsanity.com/paper/1603.05959/) (2016)
- [Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing](https://mlsanity.com/paper/2301.04558/) (2023)
- [On the Challenges and Perspectives of Foundation Models for Medical Image Analysis](https://mlsanity.com/paper/2306.05705/) (2023)
- [Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images](https://mlsanity.com/paper/2306.07831/) (2023)
- [Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision](https://mlsanity.com/paper/2310.18689/) (2023)
- [AI-based Reconstruction for Fast MRI -- A Systematic Review and Meta-analysis](https://mlsanity.com/paper/2112.12744/) (2021)
- [Brain Co-Processors: Using AI to Restore and Augment Brain Function](https://mlsanity.com/paper/2012.03378/) (2020)

## Built on by (most cited first, 30 of 39)

- [A Comprehensive Survey of Foundation Models in Medicine](https://mlsanity.com/paper/2406.10729/) (2024)
- [Triad: Vision Foundation Model for 3D Magnetic Resonance Imaging](https://mlsanity.com/paper/2502.14064/) (2025)
- [Brain Foundation Models: A Survey on Advancements in Neural Signal Processing and Brain Discovery](https://mlsanity.com/paper/2503.00580/) (2025)
- [Raptor: Scalable Train-Free Embeddings for 3D Medical Volumes Leveraging Pretrained 2D Foundation Models](https://mlsanity.com/paper/2507.08254/) (2025)
- [Building a General SimCLR Self-Supervised Foundation Model Across Neurological Diseases to Advance 3D Brain MRI Diagnoses](https://mlsanity.com/paper/2509.10620/) (2025)
- [Brain Foundation Models with Hypergraph Dynamic Adapter for Brain Disease Analysis](https://mlsanity.com/paper/2505.00627/) (2025)
- [Decipher-MR: A Vision-Language Foundation Model for 3D MRI Representations](https://mlsanity.com/paper/2509.21249/) (2025)
- [Towards General Text-guided Image Synthesis for Customized Multimodal Brain MRI Generation](https://mlsanity.com/paper/2409.16818/) (2024)
- [Conformal Lesion Segmentation for 3D Medical Images](https://mlsanity.com/paper/2510.17897/) (2025)
- [On the Utility of Foundation Models for Fast MRI: Vision-Language-Guided Image Reconstruction](https://mlsanity.com/paper/2511.19641/) (2025)
- [Large-scale Multi-sequence Pretraining for Generalizable MRI Analysis in Versatile Clinical Applications](https://mlsanity.com/paper/2508.07165/) (2025)
- [OmniMRI: A Unified Vision--Language Foundation Model for Generalist MRI Interpretation](https://mlsanity.com/paper/2508.17524/) (2025)
- [Generalist Models in Medical Image Segmentation: A Survey and Performance Comparison with Task-Specific Approaches](https://mlsanity.com/paper/2506.10825/) (2025)
- [Learning Clinical Representations Under Systematic Distribution Shift](https://mlsanity.com/paper/2603.07348/) (2026)
- [Discriminative Representation Learning for Clinical Prediction](https://mlsanity.com/paper/2603.20921/) (2026)
- [Multi-encoder nnU-Net outperforms transformer models with self-supervised pretraining](https://mlsanity.com/paper/2504.03474/) (2025)
- [Bridging Brain with Foundation Models through Self-Supervised Learning](https://mlsanity.com/paper/2506.16009/) (2025)
- [FoundBioNet: A Foundation-Based Model for IDH Genotyping of Glioma from Multi-Parametric MRI](https://mlsanity.com/paper/2508.06756/) (2025)
- [A Structured Review and Quantitative Profiling of Public Brain MRI Datasets for Foundation Model Development](https://mlsanity.com/paper/2510.20196/) (2025)
- [Statistical Opportunities in Neuroimaging](https://mlsanity.com/paper/2602.12974/) (2026)
- [BrainG3N: A Dual-Purpose Tokenizer for Controllable 3D Brain MRI Generation](https://mlsanity.com/paper/2606.19651/) (2026)
- [Vision Foundation Models in Radiology: A Scoping Review of Data, Methodology, Evaluation and Clinical Translation](https://mlsanity.com/paper/2607.07219/) (2026)
- [Site Is Decodable Before Pretraining: Negative Controls for Probing Frozen Brain-MRI Foundation Models](https://mlsanity.com/paper/2608.10295/) (2026)
- [Towards a Multimodal MRI-Based Foundation Model for Multi-Level Feature Exploration in Segmentation, Molecular Subtyping, and Grading of Glioma](https://mlsanity.com/paper/2503.06828/) (2025)
- [Brain Imaging Foundation Models, Are We There Yet? A Systematic Review of Foundation Models for Brain Imaging and Biomedical Research](https://mlsanity.com/paper/2506.13306/) (2025)
- [Converting T1-weighted MRI from 3T to 7T quality using deep learning](https://mlsanity.com/paper/2507.13782/) (2025)
- [Towards Self-Supervised Foundation Models for Critical Care Time Series](https://mlsanity.com/paper/2509.19885/) (2025)
- [Redundancy as a Structural Information Principle for Learning and Generalization](https://mlsanity.com/paper/2510.10938/) (2025)
- [Computational Imaging Meets LLMs: Zero-Shot IDH Mutation Prediction in Brain Gliomas](https://mlsanity.com/paper/2511.03376/) (2025)
- [Foundation Models for Medical Imaging: Status, Challenges, and Directions](https://mlsanity.com/paper/2602.15913/) (2026)

## Abstract (arXiv)

The burgeoning field of brain health research increasingly leverages artificial intelligence (AI) to interpret and analyze neurological data. This study introduces a novel approach towards the creation of medical foundation models by integrating a large-scale multi-modal magnetic resonance imaging (MRI) dataset derived from 41,400 participants in its own. Our method involves a novel two-stage pretraining approach using vision transformers. The first stage is dedicated to encoding anatomical structures in generally healthy brains, identifying key features such as shapes and sizes of different brain regions. The second stage concentrates on spatial information, encompassing aspects like location and the relative positioning of brain structures. We rigorously evaluate our model, BrainFounder, using the Brain Tumor Segmentation (BraTS) challenge and Anatomical Tracings of Lesions After Stroke v2.0 (ATLAS v2.0) datasets. BrainFounder demonstrates a significant performance gain, surpassing the achievements of the previous winning solutions using fully supervised learning. Our findings underscore the impact of scaling up both the complexity of the model and the volume of unlabeled training data derived from generally healthy brains, which enhances the accuracy and predictive capabilities of the model in complex neuroimaging tasks with MRI. The implications of this research provide transformative insights and practical applications in healthcare and make substantial steps towards the creation of foundation models for Medical AI. Our pretrained models and training code can be found at https://github.com/lab-smile/GatorBrain.