# Pillar-0: A New Frontier for Radiology Foundation Models

arXiv: https://arxiv.org/abs/2511.17803 · HTML: https://mlsanity.com/paper/2511.17803/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 53 references that are in the map.
- First posted on arXiv 21 Nov 2025.

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023)
- [SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering](https://mlsanity.com/paper/2102.09542/) (2021)
- [MedGemma Technical Report](https://mlsanity.com/paper/2507.05201/) (2025)
- [PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering](https://mlsanity.com/paper/2305.10415/) (2023)
- [Med3D: Transfer Learning for 3D Medical Image Analysis](https://mlsanity.com/paper/1904.00625/) (2019)
- [Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning](https://mlsanity.com/paper/2506.07044/) (2025)
- [MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding](https://mlsanity.com/paper/2501.18362/) (2025)
- [Unified Contrastive Learning in Image-Text-Label Space](https://mlsanity.com/paper/2204.03610/) (2022)
- [OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM](https://mlsanity.com/paper/2402.09181/) (2024)
- [DaViT: Dual Attention Vision Transformers](https://mlsanity.com/paper/2204.03645/) (2022)
- [OmniVL:One Foundation Model for Image-Language and Video-Language Tasks](https://mlsanity.com/paper/2209.07526/) (2022)
- [Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understanding](https://mlsanity.com/paper/2304.06906/) (2023)
- [Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset](https://mlsanity.com/paper/2406.06512/) (2024)
- [MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging](https://mlsanity.com/paper/2410.06542/) (2024)
- [Atlas: Multi-Scale Attention Improves Long Context Image Modeling](https://mlsanity.com/paper/2503.12355/) (2025)
- [Automated Real-time Assessment of Intracranial Hemorrhage Detection AI Using an Ensembled Monitoring Model (EMM)](https://mlsanity.com/paper/2505.11738/) (2025)
- [Radiology's Last Exam (RadLE): Benchmarking Frontier Multimodal AI Against Human Experts and a Taxonomy of Visual Reasoning Errors in Radiology](https://mlsanity.com/paper/2509.25559/) (2025)

## Built on by (most cited first, 25 of 25)

- [Curia-2: Scaling Self-Supervised Learning for Radiology Foundation Models](https://mlsanity.com/paper/2604.01987/) (2026)
- [TotalFM: An Organ-Separated 3D-CT Foundation Model Leveraging Large-Scale Routine Clinical Radiology Data](https://mlsanity.com/paper/2601.00260/) (2026)
- [Jolia: Concept-Level Vision-Language Alignment for 3D CT Contrastive Learning](https://mlsanity.com/paper/2606.24570/) (2026)
- [Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT](https://mlsanity.com/paper/2603.02026/) (2026)
- [Enhancing Fine-Grained Spatial Grounding in 3D CT Report Generation via Discriminative Guidance](https://mlsanity.com/paper/2604.10437/) (2026)
- [GreenRFM: Learning a resource-efficient radiology vision-language foundation model via supervision-centric pre-training](https://mlsanity.com/paper/2603.06467/) (2026)
- [MedPruner: Training-Free Hierarchical Token Pruning for Efficient 3D Medical Image Understanding in Vision-Language Models](https://mlsanity.com/paper/2603.11625/) (2026)
- [Pan-FM: A Pan-Organ Foundation Model with Saliency-Guided Masking for Missing Robustness](https://mlsanity.com/paper/2605.07055/) (2026)
- [CT-IDP: Segmentation-Derived Quantitative Phenotypes for Interpretable Abdominal CT Disease Classification](https://mlsanity.com/paper/2605.09002/) (2026)
- [JANUS: Anatomy-Conditioned Gating for Robust CT Triage Under Distribution Shift](https://mlsanity.com/paper/2605.13813/) (2026)
- [ORACLE-CT: Anatomy-Aware Support Pooling for CT Classification](https://mlsanity.com/paper/2606.05460/) (2026)
- [Open-Ended CT Volume Segmentation with Weak Supervision from Language](https://mlsanity.com/paper/2607.25860/) (2026)
- [Big, Bright, or Invisible: A Frozen-Feature Benchmark of 3D CT Foundation Models](https://mlsanity.com/paper/2608.05960/) (2026)
- [Specializing Foundation Models via Mixture of Low-Rank Experts for Comprehensive Head CT Analysis](https://mlsanity.com/paper/2603.00675/) (2026)
- [Blinded Radiologist and LLM-Based Evaluation of LLM-Generated Japanese Translations of Chest CT Reports: Comparative Study](https://mlsanity.com/paper/2604.02207/) (2026)
- [UCSF-PDGM-VQA: Visual Question Answering dataset for brain tumor MRI interpretation](https://mlsanity.com/paper/2605.17140/) (2026)
- [MedSyn2: Flexible Control of 3D CT Generation via Text and Semantically-Defined Segmentation Prompts](https://mlsanity.com/paper/2606.00967/) (2026)
- [Stateful Visual Encoders for Vision-Language Models](https://mlsanity.com/paper/2606.04433/) (2026)
- [When is 3D Worth It? A Resource-Performance Frontier for CNNs and Transformers in Lung CT](https://mlsanity.com/paper/2606.06950/) (2026)
- [A Multi-Center Benchmark for Abdominal Disease Diagnosis and Report Generation from Non-Contrast CT](https://mlsanity.com/paper/2606.16991/) (2026)
- [Curia-MAE: Multi-Modal Multi-Anatomy MAE Pre-Training for 3D Medical Image Segmentation](https://mlsanity.com/paper/2608.05844/) (2026)
- [Longitudinal 3D Foundation Modeling for Neoadjuvant Breast Cancer Response Prediction from Serial DCE-MRI](https://mlsanity.com/paper/2608.09991/) (2026)
- [NV-Reason-CT: 3D Visual Language Model for CT Analysis](https://mlsanity.com/paper/2609.27511/) (2026)
- [FD-AA: A Lightweight Focal-Diffuse And Attenuation-Aware Head for Incidental Abdominal Abnormality Detection in Chest CT](https://mlsanity.com/paper/2609.36189/) (2026)
- [Scaling 3D Visual Grounding in Abdominal CT](https://mlsanity.com/paper/2610.04095/) (2026)

## Abstract (arXiv)

Radiology plays an integral role in modern medicine, yet rising imaging volumes have far outpaced workforce growth. Foundation models offer a path toward assisting with the full spectrum of radiology tasks, but existing medical models remain limited: they process volumetric CT and MRI as low-fidelity 2D slices, discard critical grayscale contrast information, and lack evaluation frameworks that reflect real clinical practice. We introduce Pillar-0, a radiology foundation model pretrained on 42,990 abdomen-pelvis CTs, 86,411 chest CTs, 14,348 head CTs, and 11,543 breast MRIs from a large academic center, together with RATE, a scalable framework that extracts structured labels for 366 radiologic findings with near-perfect accuracy using LLMs. Across internal test sets of 14,230 abdomen-pelvis CTs, 10,646 chest CTs, 4,906 head CTs, and 1,585 breast MRIs, Pillar-0 establishes a new performance frontier, achieving mean AUROCs of 86.4, 88.0, 90.1, and 82.9, outperforming MedGemma (Google), MedImageInsight (Microsoft), Lingshu (Alibaba), and Merlin (Stanford) by 7.8-15.8 AUROC points and ranking best in 87.2% (319/366) tasks. Pillar-0 similarly outperforms all baselines in an external validation on the Stanford Abdominal CT dataset, including Merlin (82.2 vs 80.6 AUROC). Pillar-0 extends to tasks beyond its pretraining, such as long-horizon lung cancer risk prediction, where it improves upon the state-of-the-art Sybil by 3.0 C-index points on NLST, and generalizes with gains of 5.9 (MGH) and 1.9 (CGMH). In brain hemorrhage detection, Pillar-0 obtained a >95 AUROC when using only 1/20th of the data of the next most sample efficient baseline. Pillar-0 and RATE together provide an open, clinically rigorous foundation for building high-performance radiology systems, enabling applications that were previously infeasible due to computational, data, and evaluation constraints.