mlsanityData to 14 Oct 2026

Localization Lens for Improving Medical Vision-Language Models

arXiv 2610.04502 ↗First posted 3 Oct 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Medical Vision-Language Models (Med-VLMs) have demonstrated strong capabilities in clinical tasks. However, they often struggle to understand anatomical structures and spatial positioning, which are crucial for medical reasoning. To address this, we propose a localization-aware enhancement to the Med-VLM pipeline, introducing improvements at three levels: data,architecture, and alignment. First, we introduce localization lens, a set of expert-validated representations that provide richer anatomical and positional context. However, as these representations increase input complexity, we integrate pixel shuffle within the model architecture to filter and refine representations, enhancing spatial information processing while preserving anatomical continuity. Lastly, to effectively align the localization lens representations with textual features, we incorporate decoupled contrastive loss (DCL) alongside the standard loss function. This ensures better feature discrimination and robustness, particularly in data limited medical settings. Through extensive evaluations on medical visual question answering (Med-VQA) datasets, we show that our methodology improves localization-driven performance across different Med-VLM architectures. Our analysis of localization-based questions further reveals that improvements in anatomy and spatial reasoning directly enhance the overall accuracy of Med-VQA upto 6.2%. The proposed approach is model-agnostic and can be seamlessly integrated into existing Med-VLM pipelines. The dataset, code, and trained models will be made publicly available at https://github.com/CVLABLUMS/localizationlens.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 29 references that are in the map. First posted on arXiv 3 Oct 2026.

Builds on

15 of 29 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Few-shot Adaptation of Medical Vision-Language Models5 Sep
  2. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models16 Apr
  3. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review4 Mar
  4. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 11

  1. 2023Segment Anythingcited by 8,234
  2. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  3. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  4. 2021Decoupled Contrastive Learningcited by 143
  5. 2023Interactive and Explainable Region-guided Radiology Report Generationcited by 140
  6. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 131
  7. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 92
  8. 2024When Do We Not Need Larger Vision Models?cited by 67
  9. 2023Are Natural Domain Foundation Models Useful for Medical Image Classification?cited by 46
  10. 2024ChEX: Interactive Localization and Region Description in Chest X-rayscited by 19
  11. 2024Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Reasoning to Pixels: Grounded Medical Multimodal LLMs for VQA and Segmentation
  2. 2026Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models
  3. 2025MedCLM: Learning to Localize and Reason via a CoT-Curriculum in Medical Vision-Language Models
  4. 2026Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Space
  5. 2026Lost in Volume: The CT-SpatialVQA Benchmark for Evaluating Semantic-Spatial Understanding of 3D Medical Vision-Language Models
  6. 2026EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding
  7. 2026MedPixel: A Unified Pixel-Language Model for Medical Reasoning and Segmentation
  8. 2026MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models