# Neuralizer: General Neuroimage Analysis without Re-Training

arXiv: https://arxiv.org/abs/2305.02644 · HTML: https://mlsanity.com/paper/2305.02644/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 125 references that are in the map.
- First posted on arXiv 4 May 2023.

## Builds on (most cited first, 30 of 55)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [VoxelMorph: A Learning Framework for Deformable Medical Image Registration](https://mlsanity.com/paper/1809.05231/) (2018)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019)
- [Transfusion: Understanding Transfer Learning for Medical Imaging](https://mlsanity.com/paper/1902.07208/) (2019)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Med3D: Transfer Learning for 3D Medical Image Analysis](https://mlsanity.com/paper/1904.00625/) (2019)
- [Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/1906.00817/) (2019)
- [Few-Shot Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1905.01723/) (2019)
- [CANet: Class-Agnostic Segmentation Networks with Iterative Refinement and Attentive Few-Shot Learning](https://mlsanity.com/paper/1903.02351/) (2019)
- [Adaptive Prototype Learning and Allocation for Few-Shot Segmentation](https://mlsanity.com/paper/2104.01893/) (2021)
- [Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders](https://mlsanity.com/paper/1812.01784/) (2018)
- [What makes ImageNet good for transfer learning?](https://mlsanity.com/paper/1608.08614/) (2016)
- [FSS-1000: A 1000-Class Dataset for Few-Shot Segmentation](https://mlsanity.com/paper/1907.12347/) (2019)
- [SynthMorph: learning contrast-invariant registration without acquired images](https://mlsanity.com/paper/2004.10282/) (2020)
- [Data augmentation using learned transformations for one-shot medical image segmentation](https://mlsanity.com/paper/1902.09383/) (2019)
- [SynthStrip: Skull-Stripping for Any Brain Image](https://mlsanity.com/paper/2203.09974/) (2022)
- [Multi-site fMRI Analysis Using Privacy-preserving Federated Learning and Domain Adaptation: ABIDE Results](https://mlsanity.com/paper/2001.05647/) (2020)
- [Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities and Results of the WMH Segmentation Challenge](https://mlsanity.com/paper/1904.00682/) (2019)
- [ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset](https://mlsanity.com/paper/2206.06694/) (2022)
- [nnU-Net for Brain Tumor Segmentation](https://mlsanity.com/paper/2011.00848/) (2020)

## Built on by (most cited first, 24 of 24)

- [ScribblePrompt: Fast and Flexible Interactive Segmentation for Any Biomedical Image](https://mlsanity.com/paper/2312.07381/) (2023)
- [Position: The Future of Bayesian Prediction Is Prior-Fitted](https://mlsanity.com/paper/2505.23947/) (2025)
- [Show and Segment: Universal Medical Image Segmentation via In-Context Learning](https://mlsanity.com/paper/2503.19359/) (2025)
- [A Foundation Model for Brain Lesion Segmentation with Mixture of Modality Experts](https://mlsanity.com/paper/2405.10246/) (2024)
- [MultiverSeg: Scalable Interactive Segmentation of Biomedical Imaging Datasets with In-Context Guidance](https://mlsanity.com/paper/2412.15058/) (2024)
- [MultiMorph: On-demand Atlas Construction](https://mlsanity.com/paper/2504.00247/) (2025)
- [VoxelPrompt: A Vision Agent for End-to-End Medical Image Analysis](https://mlsanity.com/paper/2410.08397/) (2024)
- [Conquering the Retina: Bringing Visual in-Context Learning to OCT](https://mlsanity.com/paper/2506.15200/) (2025)
- [Is Visual in-Context Learning for Compositional Medical Tasks within Reach?](https://mlsanity.com/paper/2507.00868/) (2025)
- [MedSAMix: A Training-Free Model Merging Approach for Medical Image Segmentation](https://mlsanity.com/paper/2508.11032/) (2025)
- [HyperCoil-Recon: A Hypernetwork-based Adaptive Coil Configuration Task Switching Network for MRI Reconstruction](https://mlsanity.com/paper/2308.04821/) (2023)
- [Towards Robust In-Context Learning for Medical Image Segmentation via Data Synthesis](https://mlsanity.com/paper/2509.19711/) (2025)
- [Neuroverse3D: Developing In-Context Learning Universal Model for Neuroimaging in 3D](https://mlsanity.com/paper/2503.02410/) (2025)
- [Efficient Universal Models for Medical Image Segmentation via Weakly Supervised In-Context Learning](https://mlsanity.com/paper/2510.05899/) (2025)
- [Deep infant brain segmentation from multi-contrast MRI](https://mlsanity.com/paper/2512.05114/) (2025)
- [From Static to Interactive: Adapting Visual in-Context Learners for User-Driven Tasks](https://mlsanity.com/paper/2604.06748/) (2026)
- [GMISeg: General Medical Image Segmentation without Re-Training](https://mlsanity.com/paper/2311.12539/) (2023)
- [Probing Intrinsic Medical Task Relationships: A Contrastive Learning Perspective](https://mlsanity.com/paper/2604.05651/) (2026)
- [CoRE: Concept-Reasoning Expansion for Continual Brain Lesion Segmentation](https://mlsanity.com/paper/2604.25376/) (2026)
- [Beyond Model Size: Probing the Gaps in Visual in-Context Learning by Training a Tiny Model](https://mlsanity.com/paper/2606.10905/) (2026)
- [Quo Vadis, Visual In-Context Learning? A Unified Benchmark Across Domains and Tasks](https://mlsanity.com/paper/2606.10967/) (2026)
- [Structural MRI Synthesis for Alzheimer's Disease via Conditional Diffusion on Anatomical Masks](https://mlsanity.com/paper/2606.18354/) (2026)
- [UniMedSeg: Unified In-Context Learning for Multi-Paradigm 2D/3D Medical Image Segmentation](https://mlsanity.com/paper/2607.12896/) (2026)
- [SHFormer: Dynamic Spectral Filtering Convolutional Neural Network and High-pass Kernel Generation Transformer for Adaptive MRI Reconstruction](https://mlsanity.com/paper/2607.20159/) (2026)

## Abstract (arXiv)

Neuroimage processing tasks like segmentation, reconstruction, and registration are central to the study of neuroscience. Robust deep learning strategies and architectures used to solve these tasks are often similar. Yet, when presented with a new task or a dataset with different visual characteristics, practitioners most often need to train a new model, or fine-tune an existing one. This is a time-consuming process that poses a substantial barrier for the thousands of neuroscientists and clinical researchers who often lack the resources or machine-learning expertise to train deep learning models. In practice, this leads to a lack of adoption of deep learning, and neuroscience tools being dominated by classical frameworks. We introduce Neuralizer, a single model that generalizes to previously unseen neuroimaging tasks and modalities without the need for re-training or fine-tuning. Tasks do not have to be known a priori, and generalization happens in a single forward pass during inference. The model can solve processing tasks across multiple image modalities, acquisition methods, and datasets, and generalize to tasks and modalities it has not been trained on. Our experiments on coronal slices show that when few annotated subjects are available, our multi-task network outperforms task-specific baselines without training on the task.