mlsanityData to 7 Oct 2026

Pretraining of Medical Visual Encoders Toward Multi-modal Large Language Models

arXiv 2609.23860 ↗First posted 20 Sep 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning also fits: Medical vision-language models

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) commonly reuse visual encoders pretrained with CLIP, although the features of these ViTs are ultimately consumed by autoregressive LLMs. We refer to this mismatch as the semantic-interface gap and introduce MedMLIP, a framework that pretrains the visual encoder through report generation with a frozen LLM, while employing Local Relational Distillation (LRD) to preserve relationships among visual patches to avoid visual collapse. We pretrain MedMLIP on IU-Xray and Open-PMC-300K and evaluate the resulting encoders on VQA-RAD and SLAKE. Only the ViT is transferred, while the guiding LLM and projector are replaced, allowing us to assess cross-LLM transferability. Our cross-LLM transfer experiments demonstrate the value of pretraining visual encoders for their autoregressive LLM interface while trying to preserve more fine-grained visual information. Code and the pretrained model are available at https://github.com/SkyCol/MedMLIP

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 12 of its 15 references that are in the map. First posted on arXiv 20 Sep 2026.

Builds on

12 of 15 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep

Other references · 11

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  8. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  9. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  10. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  11. 2025Advancing Medical Representation Learning Through High-Quality Datacited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
  2. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training
  3. 2025QwenCLIP: Boosting Medical Vision-Language Pretraining via LLM Embeddings and Prompt tuning
  4. 2026VIVID-Med: LLM-Supervised Structured Pretraining for Deployable Medical ViTs
  5. 2025OmniV-Med: Scaling Medical Vision-Language Model for Universal Visual Understanding
  6. 2025Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMs
  7. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  8. 2025Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Models