# Learning Anatomy-Grounded CT Vision-Language Representations with Organ-Hierarchical Report Knowledge

arXiv: https://arxiv.org/abs/2607.10953 · HTML: https://mlsanity.com/paper/2607.10953/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 12 Jul 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography](https://mlsanity.com/paper/2403.17834/) (2024)
- [Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition](https://mlsanity.com/paper/1708.07632/) (2017)
- [Joint Learning of Localized Representations from Medical Images and Reports](https://mlsanity.com/paper/2112.02889/) (2021)
- [Machine-Learning-Based Multiple Abnormality Prediction with Large-Scale Chest Computed Tomography Volumes](https://mlsanity.com/paper/2002.04752/) (2020)
- [Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding](https://mlsanity.com/paper/2501.14548/) (2025)
- [CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarios](https://mlsanity.com/paper/2404.15272/) (2024)
- [Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Models](https://mlsanity.com/paper/2404.04936/) (2024)
- [Comprehensive language-image pre-training for 3D medical image understanding](https://mlsanity.com/paper/2510.15042/) (2025)
- [Bridged Semantic Alignment for Zero-shot 3D Medical Image Diagnosis](https://mlsanity.com/paper/2501.03565/) (2025)
- [MG-3D: Multi-Grained Knowledge-Enhanced 3D Medical Vision-Language Pre-training](https://mlsanity.com/paper/2412.05876/) (2024)
- [BCNet: Bronchus Classification via Structure Guided Representation Learning](https://mlsanity.com/paper/2205.06947/) (2022)

## Built on by (most cited first, 2 of 2)

- [OrganLens: Organ-Specific Representation Learning for CT Foundation Models](https://mlsanity.com/paper/2607.25164/) (2026)
- [Auditable agentic AI for evidence-grounded thyroid ultrasound diagnosis and reporting](https://mlsanity.com/paper/2608.12590/) (2026)

## Abstract (arXiv)

Medical vision-language pretraining (VLP) from paired CT images and radiology reports enables scalable representation learning, but most existing methods align either whole scans with entire reports or local image regions with text fragments. These formulations underuse a key property of radiology reports: findings are organized around anatomical structures, with abnormalities described by organs, disease concepts, locations, and severity-related attributes. We propose OKA-CT, an organ-hierarchical knowledge-augmented framework for CT-report VLP. OKA-CT first converts free-text reports into organ-conditioned knowledge using radiology report parsing and LLM-assisted semantic structuring. The extracted hierarchy is used across two learning stages. Stage~1 injects anatomy-grounded evidence into the CT visual representation through fine-grained organ-conditioned supervision, while Stage~2 uses organ-specific report evidence to guide structured report-CT contrastive learning, where hierarchy-derived semantic soft targets treat non-paired cases with shared organ-level findings as weak semantic positives rather than uniform negatives. A lightweight query-based global branch further aggregates disease-relevant volumetric evidence for whole-scan representation. On CT-RATE and RAD-ChestCT datasets, OKA-CT achieves zero-shot abnormality diagnosis AUROCs of 84.9 and 72.2, outperforming prior CT VLP baselines. Retrieval and patch-occlusion analyses further show improved report-image alignment and stronger sensitivity to disease-associated anatomical regions.