mlsanityData to 7 Oct 2026

RadPRISM: Schema-stratified radiology-report supervision for concept-disentangled image representations and visual grounding

arXiv 2608.00147 ↗First posted 31 Jul 2026cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-language pretraining learns rich medical image representations from radiology reports, but previous model variants commonly operate within a single shared embedding space, so concept-level structure and interpretability must be recovered post hoc, limiting model transparency and, hence, clinical utility. We introduce RadPRISM, which makes a clinician-defined radiology schema a designated stratification axis: an on-premise large language model extracts per-concept text spans from free-text reports, and each clinical concept is aligned in its own dedicated visual subspace, turning concept stratification into direct, top-level alignment supervision. Instantiated on chest radiographs with a 19-concept schema over 203,602 examinations from an internal multi-year archive, RadPRISM improved internal dataset zero-shot classification from 0.717 (95% CI, 0.710-0.723) to 0.868 (95% CI, 0.863-0.872) macro AUROC over a matched global-alignment baseline, performed on par with the purpose-built CARZero reference in external zero-shot classification while substantially outperforming it (up to 4.3-fold) in pointing-game visual grounding. In addition, a radiologist reader study demonstrated concept-stratified retrieval ability (0.78 macro retrieval correctness rate within rank 3), surfacing disentangled descriptive findings that report-level retrieval and fixed-label vocabularies cannot express. RadPRISM yields discriminative, spatially faithful, natively concept-stratified representations shaped by and transparently inspectable by clinicians.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 46 references that are in the map. First posted on arXiv 31 Jul 2026.

Builds on

20 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Anatomy-VLM: A Fine-grained Vision-Language Model for Medical Interpretation11 Nov
  2. 2025XBench: A Comprehensive Benchmark for Visual-Language Explanations in Chest Radiography22 Oct

Other references · 18

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  4. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  5. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  6. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  7. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  8. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  9. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  10. 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 192
  11. 2024MAIRA-2: Grounded Radiology Report Generationcited by 140
  12. 2020Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generationcited by 139
6 more
  1. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  2. 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
  3. 2024CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classificationcited by 40
  4. 2021How I failed machine learning in medical imaging -- shortcomings and recommendationscited by 37
  5. 2024CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarioscited by 29
  6. 2025RadAlign: Advancing Radiology Report Generation with Vision-Language Concept Alignmentcited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Visual Alignment of Medical Vision-Language Models for Grounded Radiology Report Generation
  2. 2026RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture
  3. 2026Harrison.Rad 1.5 Technical Report: A radiology foundation model that can draft reports from images, priors and clinical context
  4. 2026GreenRFM: Learning a resource-efficient radiology vision-language foundation model via supervision-centric pre-training
  5. 2026Graph-Supervised Hierarchical Clinical Alignment for Radiology Report Generation with Large Language Models
  6. 2026AlphaRAD: Grounded Zero-Shot Classification in Chest Radiology via α-Corrected Binary Cross Entropy and Factorized Latent Supervision
  7. 2025A Reality Check of Vision-Language Pre-training in Radiology: Have We Progressed Using Text?
  8. 2026RadAnnotate: Large Language Models for Efficient and Reliable Radiology Report Annotation