# Jolia: Concept-Level Vision-Language Alignment for 3D CT Contrastive Learning

arXiv: https://arxiv.org/abs/2606.24570 · HTML: https://mlsanity.com/paper/2606.24570/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 23 Jun 2026.

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning](https://mlsanity.com/paper/2210.06044/) (2022)
- [GREEN: Generative Radiology Report Evaluation and Error Notation](https://mlsanity.com/paper/2405.03595/) (2024)
- [Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding](https://mlsanity.com/paper/2501.14548/) (2025)
- [MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging](https://mlsanity.com/paper/2410.06542/) (2024)
- [Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysis](https://mlsanity.com/paper/2503.20047/) (2025)
- [CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarios](https://mlsanity.com/paper/2404.15272/) (2024)
- [INSPECT: A Multimodal Dataset for Pulmonary Embolism Diagnosis and Prognosis](https://mlsanity.com/paper/2311.10798/) (2023)
- [Pillar-0: A New Frontier for Radiology Foundation Models](https://mlsanity.com/paper/2511.17803/) (2025)
- [RadEval: A framework for radiology text evaluation](https://mlsanity.com/paper/2509.18030/) (2025)
- [Comprehensive language-image pre-training for 3D medical image understanding](https://mlsanity.com/paper/2510.15042/) (2025)
- [Curia: A Multi-Modal Foundation Model for Radiology](https://mlsanity.com/paper/2509.06830/) (2025)
- [CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation](https://mlsanity.com/paper/2603.06183/) (2026)
- [Scaling Self-Supervised and Cross-Modal Pretraining for Volumetric CT Transformers](https://mlsanity.com/paper/2511.17209/) (2025)
- [RadSAM: Segmenting 3D radiological images with a 2D promptable model](https://mlsanity.com/paper/2504.20837/) (2025)
- [Curia-2: Scaling Self-Supervised Learning for Radiology Foundation Models](https://mlsanity.com/paper/2604.01987/) (2026)
- [Atlas: Multi-Scale Attention Improves Long Context Image Modeling](https://mlsanity.com/paper/2503.12355/) (2025)
- [TotalFM: An Organ-Separated 3D-CT Foundation Model Leveraging Large-Scale Routine Clinical Radiology Data](https://mlsanity.com/paper/2601.00260/) (2026)

## Built on by (most cited first, 4 of 4)

- [Resolution Meets Reduction: Efficient Visual Context for 3D Radiology Report Generation](https://mlsanity.com/paper/2608.08713/) (2026)
- [Volumetric Radiology AI in the Era of Multimodal Large Language Models](https://mlsanity.com/paper/2608.20549/) (2026)
- [NV-Reason-CT: 3D Visual Language Model for CT Analysis](https://mlsanity.com/paper/2609.27511/) (2026)
- [FD-AA: A Lightweight Focal-Diffuse And Attenuation-Aware Head for Incidental Abdominal Abnormality Detection in Chest CT](https://mlsanity.com/paper/2609.36189/) (2026)

## Abstract (arXiv)

Vision-language contrastive pretraining has become the dominant recipe for 3D medical foundation models, leveraging the large volumes of paired scans and reports produced in clinical practice. However, medical images usually span dozens of organs, and radiological reports are much longer than typical natural image captions and are composed of multiple structured sections. CLIP-style pretraining compresses this structure by encoding each modality into a single global token, at the risk of losing important details. We introduce ConQuer (Concept Queries), an image-text pretraining method that augments CLIP's global alignment with a set of localized alignments, one per concept. ConQuer splits the report into concept-specific sections and learns cross-attention queries that pool the matching image features without using any segmentation mask or spatial supervision. Contrastive learning is then applied independently for each concept. Concepts can be any unit of semantic localization; here, they are anatomical regions, one query per organ or gross body region. As a byproduct, each query learns attention maps focused on its concept, providing built-in spatial interpretability. We use ConQuer to train Jolia, a 3D CT foundation model on chest and abdominal CT. Jolia consistently outperforms a CLIP baseline on findings classification, report generation, and cross-center transfer, and sets a new state of the art across multiple public benchmarks. Jolia's weights are available at https://huggingface.co/raidium/Jolia