# MedKCO: Medical Vision-Language Pretraining via Knowledge-Driven Cognitive Orchestration

arXiv: https://arxiv.org/abs/2603.09101 · HTML: https://mlsanity.com/paper/2603.09101/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 27 of its 43 references that are in the map.
- First posted on arXiv 10 Mar 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographs](https://mlsanity.com/paper/1910.03667/) (2019)
- [CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training](https://mlsanity.com/paper/2310.13292/) (2023)
- [A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision](https://mlsanity.com/paper/2308.07898/) (2023)
- [COVIDX-Net: A Framework of Deep Learning Classifiers to Diagnose COVID-19 in X-Ray Images](https://mlsanity.com/paper/2003.11055/) (2020)
- [Applications of Deep Learning in Fundus Images: A Review](https://mlsanity.com/paper/2101.09864/) (2021)
- [A Multimodal Vision Foundation Model for Clinical Dermatology](https://mlsanity.com/paper/2410.15038/) (2024)
- [OCTID: Optical Coherence Tomography Image Database](https://mlsanity.com/paper/1812.07056/) (2018)
- [CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Grounding](https://mlsanity.com/paper/2305.08685/) (2023)
- [OCTDL: Optical Coherence Tomography Dataset for Image-Based Deep Learning Methods](https://mlsanity.com/paper/2312.08255/) (2023)
- [Derm1M: A Million-scale Vision-Language Dataset Aligned with Clinical Ontology Knowledge for Dermatology](https://mlsanity.com/paper/2503.14911/) (2025)
- [MM-Retinal: Knowledge-Enhanced Foundational Pretraining with Fundus Image-Text Expertise](https://mlsanity.com/paper/2405.11793/) (2024)
- [Learning Robust Representation for Joint Grading of Ophthalmic Diseases via Adaptive Curriculum and Feature Disentanglement](https://mlsanity.com/paper/2207.04183/) (2022)
- [CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment](https://mlsanity.com/paper/2406.05205/) (2024)
- [ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation](https://mlsanity.com/paper/2311.13258/) (2023)
- [Curriculum Learning for Data-Efficient Vision-Language Alignment](https://mlsanity.com/paper/2207.14525/) (2022)
- [MM-Retinal V2: Transfer an Elite Knowledge Spark into Fundus Vision-Language Pretraining](https://mlsanity.com/paper/2501.15798/) (2025)
- [Continual Retinal Vision-Language Pre-training upon Incremental Imaging Modalities](https://mlsanity.com/paper/2506.19320/) (2025)
- [From Prototypes to General Distributions: An Efficient Curriculum for Masked Image Modeling](https://mlsanity.com/paper/2411.10685/) (2024)

## Abstract (arXiv)

Medical vision-language pretraining (VLP) models have recently been investigated for their generalization to diverse downstream tasks. However, current medical VLP methods typically force the model to learn simple and complex concepts simultaneously. This anti-cognitive process leads to suboptimal feature representations, especially under distribution shift. To address this limitation, we propose a Knowledge-driven Cognitive Orchestration for Medical VLP (MedKCO) that involves both the ordering of the pretraining data and the learning objective of vision-language contrast. Specifically, we design a two level curriculum by incorporating diagnostic sensitivity and intra-class sample representativeness for the ordering of the pretraining data. Moreover, considering the inter-class similarity of medical images, we introduce a self-paced asymmetric contrastive loss to dynamically adjust the participation of the pretraining objective. We evaluate the proposed pretraining method on three medical imaging scenarios in multiple vision-language downstream tasks, and compare it with several curriculum learning methods. Extensive experiments show that our method significantly surpasses all baselines. https://github.com/Mr-Talon/MedKCO.