# Unified Supervision For Vision-Language Modeling in 3D Computed Tomography

arXiv: https://arxiv.org/abs/2509.01554 · HTML: https://mlsanity.com/paper/2509.01554/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 1 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records](https://mlsanity.com/paper/2203.03540/) (2022)
- [Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography](https://mlsanity.com/paper/2403.17834/) (2024)
- [M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models](https://mlsanity.com/paper/2404.00578/) (2024)
- [Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Images](https://mlsanity.com/paper/2302.14042/) (2023)
- [Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset](https://mlsanity.com/paper/2406.06512/) (2024)
- [Machine-Learning-Based Multiple Abnormality Prediction with Large-Scale Chest Computed Tomography Volumes](https://mlsanity.com/paper/2002.04752/) (2020)
- [Y-Net: Joint Segmentation and Classification for Diagnosis of Breast Biopsy Images](https://mlsanity.com/paper/1806.01313/) (2018)
- [MediCLIP: Adapting CLIP for Few-shot Medical Image Anomaly Detection](https://mlsanity.com/paper/2405.11315/) (2024)
- [E3D-GPT: Enhanced 3D Visual Foundation for Medical Vision-Language Model](https://mlsanity.com/paper/2410.14200/) (2024)
- [3D-CT-GPT: Generating 3D Radiology Reports through Integration of Large Vision-Language Models](https://mlsanity.com/paper/2409.19330/) (2024)
- [Learning A Multi-Task Transformer Via Unified And Customized Instruction Tuning For Chest Radiograph Interpretation](https://mlsanity.com/paper/2311.01092/) (2023)
- [UMIT: Unifying Medical Imaging Tasks via Vision-Language Models](https://mlsanity.com/paper/2503.15892/) (2025)
- [3D Convolutional Neural Networks for Stalled Brain Capillary Detection](https://mlsanity.com/paper/2104.01687/) (2021)
- [Navigating Data Scarcity using Foundation Models: A Benchmark of Few-Shot and Zero-Shot Learning Approaches in Medical Imaging](https://mlsanity.com/paper/2408.08058/) (2024)
- [Towards Universal Text-driven CT Image Segmentation](https://mlsanity.com/paper/2503.06030/) (2025)

## Built on by (most cited first, 4 of 4)

- [Organ-Aware Attention Improves CT Triage and Classification](https://mlsanity.com/paper/2601.13385/) (2026)
- [GreenRFM: Learning a resource-efficient radiology vision-language foundation model via supervision-centric pre-training](https://mlsanity.com/paper/2603.06467/) (2026)
- [ORACLE-CT: Anatomy-Aware Support Pooling for CT Classification](https://mlsanity.com/paper/2606.05460/) (2026)
- [Efficient Multimodal Clinical Question Answering for Pulmonary Embolism Risk Assessment](https://mlsanity.com/paper/2606.22442/) (2026)

## Abstract (arXiv)

General-purpose vision-language models (VLMs) have emerged as promising tools in radiology, offering zero-shot capabilities that mitigate the need for large labeled datasets. However, in high-stakes domains like diagnostic radiology, these models often lack the discriminative precision required for reliable clinical use. This challenge is compounded by the scarcity and heterogeneity of publicly available volumetric CT datasets, which vary widely in annotation formats and granularity. To address these limitations, we introduce Uniferum, a volumetric VLM that unifies diverse supervision signals, encoded in classification labels and segmentation masks, into a single training framework. By harmonizing three public 3D CT datasets with distinct annotations, Uniferum achieves state-of-the-art performance, improving AUROC on the CT-RATE benchmark by 7% compared to CLIP-based and conventional multi-label convolutional models. The model demonstrates robust out-of-distribution generalization, with observed evidence of unexpected zero-shot performance on the RAD-CHEST and INSPECT datasets. Our results highlight the effectiveness of integrating heterogeneous annotations and body segmentation to enhance model performance, setting a new direction for clinically reliable, data-efficient VLMs in 3D medical imaging.