mlsanityData to 7 Oct 2026

Fine-Grained Vision-Language Pretraining with Organ-Conditioned Pattern Tokens for CT Understanding

arXiv 2607.13892 ↗First posted 15 Jul 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Computed tomography (CT) vision-language pretraining from paired volumes and radiology reports is a scalable yet challenging task. Existing methods commonly adopt global scan-report contrast, which is scalable but obscures heterogeneous organ evidence. Meanwhile, direct organ-level alignment remains coarse, since the same anatomy can exhibit multiple distinct radiological appearances. Therefore, pretraining requires a finer alignment unit: the organ-conditioned radiological pattern. In this work, we propose OCP-CT, an organ-conditioned pattern-token alignment framework for CT vision-language pretraining. Specifically, OCP-CT preserves a stable global CT-report contrastive branch and introduces an organ pattern interface: sparse Mixture-of-Experts (MoE) routes image and text tokens according to latent radiological patterns, learnable slots query the routed tokens into continuous pattern tokens, and paired token contrast aligns image-text pattern tokens with structured soft targets built from report-derived clinical similarity. On the publicly available CT-RATE and RAD-ChestCT benchmarks, OCP-CT achieves average AUROCs of 84.5% and 69.9% for zero-shot abnormality diagnosis, respectively. Compared with the strongest prior reported results, these results yield absolute AUROC gains of 6.7 and 0.8 percentage points.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 26 references that are in the map. First posted on arXiv 15 Jul 2026.

Builds on

23 of 26 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Comprehensive language-image pre-training for 3D medical image understanding16 Oct
  2. 2025Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding24 Jan
  3. 2024MG-3D: Multi-Grained Knowledge-Enhanced 3D Medical Vision-Language Pre-training8 Dec
  4. 2024Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset10 Jun

Other references · 19

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  8. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  9. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  10. 2020Contrastive Learning with Hard Negative Samplescited by 619
  11. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  12. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
7 more
  1. 2023From Sparse to Soft Mixtures of Expertscited by 245
  2. 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 192
  3. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  4. 2020Machine-Learning-Based Multiple Abnormality Prediction with Large-Scale Chest Computed Tomography Volumescited by 66
  5. 2024CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarioscited by 29
  6. 2024Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Modelscited by 24
  7. 2025Bridged Semantic Alignment for Zero-shot 3D Medical Image Diagnosiscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning Anatomy-Grounded CT Vision-Language Representations with Organ-Hierarchical Report Knowledge
  2. 2026Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT
  3. 2026Learning How Much, Not Just What: Cross-Patient Burden Order for CT Vision-Language Pretraining
  4. 2026Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography
  5. 2025More performant and scalable: Rethinking contrastive vision-language pre-training of radiology in the LLM era
  6. 2026Representation Learning with Semantic-aware Instance and Sparse Token Alignments
  7. 2025Boosting Vision Semantic Density with Anatomy Normality Modeling for Medical Vision-language Pre-training
  8. 2026Anatomy Contextualized Adaptation of CT Foundation Models