# A Generalist Foundation Model for Total-body PET/CT Enables Diagnostic Reporting and System-wide Metabolic Profiling

arXiv: https://arxiv.org/abs/2601.12820 · HTML: https://mlsanity.com/paper/2601.12820/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 47 references that are in the map.
- First posted on arXiv 19 Jan 2026.

## Builds on (most cited first, 24 of 24)

- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis](https://mlsanity.com/paper/1908.06912/) (2019)
- [Automated Radiology Report Generation: A Review of Recent Advances](https://mlsanity.com/paper/2405.10842/) (2024)
- [RadGPT: Constructing 3D Image-Text Tumor Datasets](https://mlsanity.com/paper/2501.04678/) (2025)
- [Dia-LLaMA: Towards Large Language Model-driven CT Report Generation](https://mlsanity.com/paper/2403.16386/) (2024)
- [Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation](https://mlsanity.com/paper/2407.02235/) (2024)
- [From FDG to PSMA: A Hitchhiker's Guide to Multitracer, Multicenter Lesion Segmentation in PET/CT Imaging](https://mlsanity.com/paper/2409.09478/) (2024)
- [Autopet III challenge: Incorporating anatomical knowledge into nnUNet for lesion segmentation in PET/CT](https://mlsanity.com/paper/2409.12155/) (2024)
- [Developing a PET/CT Foundation Model for Cross-Modal Anatomical and Functional Imaging](https://mlsanity.com/paper/2503.02824/) (2025)
- [AutoPET Challenge: Tumour Synthesis for Data Augmentation](https://mlsanity.com/paper/2409.08068/) (2024)
- [Enhancing Lesion Segmentation in PET/CT Imaging with Deep Learning and Advanced Data Preprocessing Techniques](https://mlsanity.com/paper/2409.09784/) (2024)
- [Lesion Segmentation in Whole-Body Multi-Tracer PET-CT Images; a Contribution to AutoPET 2024 Challenge](https://mlsanity.com/paper/2409.14475/) (2024)
- [An alternative formulation of attention pooling function in translation](https://mlsanity.com/paper/2409.00068/) (2024)
- [AutoPET Challenge III: Testing the Robustness of Generalized Dice Focal Loss trained 3D Residual UNet for FDG and PSMA Lesion Segmentation from Whole-Body PET/CT Images](https://mlsanity.com/paper/2409.10151/) (2024)
- [AutoPET III Challenge: PET/CT Semantic Segmentation](https://mlsanity.com/paper/2409.13006/) (2024)
- [Sine Wave Normalization for Deep Learning-Based Tumor Segmentation in CT/PET Imaging](https://mlsanity.com/paper/2409.13410/) (2024)
- [Data Diet: Can Trimming PET/CT Datasets Enhance Lesion Segmentation?](https://mlsanity.com/paper/2409.13548/) (2024)
- [AutoPET III Challenge: Tumor Lesion Segmentation using ResEnc-Model Ensemble](https://mlsanity.com/paper/2409.13779/) (2024)
- [AutoPETIII: The Tracer Frontier. What Frontier?](https://mlsanity.com/paper/2410.02807/) (2024)

## Abstract (arXiv)

Total-body PET/CT enables system-wide molecular imaging, but heterogeneous anatomical and metabolic signals, approximately 2 m axial coverage, and structured radiology semantics challenge existing medical AI models that assume single-modality inputs, localized fields of view, and coarse image-text alignment. We introduce SDF-HOLO (Systemic Dual-stream Fusion Holo Model), a multimodal foundation model for holistic total-body PET/CT, pre-trained on more than 10,000 patients. SDF-HOLO decouples CT and PET representation learning with dual-stream encoders and couples them through a cross-modal interaction module, allowing anatomical context to refine PET aggregation while metabolic saliency guides subtle morphological reasoning. To model long-range dependencies across the body, hierarchical context modeling combines efficient local windows with global attention. To bridge voxels and clinical language, we use anatomical segmentation masks as explicit semantic anchors and perform voxel-mask-text alignment during pre-training. Across tumor segmentation, low-dose lesion detection, and multilingual diagnostic report generation, SDF-HOLO outperforms strong task-specific and clinical-reference baselines while reducing localization errors and hallucinated findings. Beyond focal interpretation, the model enables system-wide metabolic profiling and reveals tumor-associated fingerprints of inter-organ metabolic network interactions, providing a scalable computational foundation for total-body PET/CT diagnostics and system-level precision oncology.