mlsanityData to 14 Oct 2026

ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Models

arXiv 2410.02615 ↗First posted 3 Oct 2024cs.LGcited by 6

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

State-of-the-art medical multi-modal LLMs (med-MLLMs), such as LLaVA-Med and BioMedGPT, primarily depend on scaling model size and data volume, with training driven largely by autoregressive objectives. However, we reveal that this approach can lead to weak vision-language alignment, making these models overly dependent on costly instruction-following data. To address this, we introduce ExGra-Med, a novel multi-graph alignment framework that jointly aligns images, instruction responses, and extended captions in the latent space, advancing semantic grounding and cross-modal coherence. To scale to large LLMs (e.g., LLaMA-7B), we develop an efficient end-to-end training scheme using black-box gradient estimation, enabling fast and scalable optimization. Empirically, ExGra-Med matches LLaVA-Med's performance using just 10% of the pre-training data, achieving a 20.13% gain on VQA-RAD and approaching full-data performance. It also outperforms strong baselines like BioMedGPT and RadFM on visual chatbot and zero-shot classification tasks, demonstrating its promise for efficient, high-quality vision-language integration in medical AI.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 64 of its 108 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

64 of 108 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine6 Aug
  2. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM14 Feb
  3. 2023LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching20 Jun
  4. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  5. 2022Self-supervised vision-language pretraining for Medical visual question answering24 Nov

Other references · 59

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
38 more
  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  3. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  4. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  5. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  6. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  7. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  8. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  9. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  10. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  11. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  12. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  13. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 384
  14. 2023Towards Generalist Biomedical AIcited by 289
  15. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  16. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 274
  17. 2020COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Imagescited by 264
  18. 2019Differentiation of Blackbox Combinatorial Solverscited by 226
  19. 2020Can AI help in screening Viral and COVID-19 pneumonia?cited by 200
  20. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 179
  21. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  22. 2024BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at oncecited by 111
  23. 2020COVID-19 Image Data Collection: Prospective Predictions Are the Futurecited by 100
  24. 2021Implicit MLE: Backpropagating Through Discrete Exponential Family Distributionscited by 98
  25. 2020Deep Graph Matching via Blackbox Differentiation of Combinatorial Solverscited by 81
  26. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  27. 2023A Comprehensive Survey on Deep Graph Representation Learningcited by 72
  28. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 62
  29. 2019Optimizing Rank-based Metrics with Blackbox Differentiationcited by 58
  30. 2024Dense Connector for MLLMscited by 49
  31. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 33
  32. 2021Wasserstein barycenters are NP-hard to computecited by 28
  33. 2016A Study of Lagrangean Decompositions and Dual Ascent Solvers for Graph Matchingcited by 26
  34. 2018Higher-order Projected Power Iterations for Scalable Multi-Matchingcited by 24
  35. 2023Self-Supervised Learning for Multimodal Non-Rigid 3D Shape Matchingcited by 23
  36. 2023Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learningcited by 16
  37. 2023Parrot: Enhancing Multi-Turn Instruction Following for Large Language Modelscited by 16
  38. 2024Partial-to-Partial Shape Matching with Geometric Consistencycited by 16

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 1.

6 papers.

  1. 2025UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysiscited by 12
  2. 2025Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMscited by 11
  3. 2025Deep Learning for Ophthalmology: The State-of-the-Art and Future Trendscited by 1
  4. 2025Quizzard@INOVA Challenge 2025 -- Track A: Plug-and-Play Technique in Interleaved Multi-Image Modelcited by 1
  5. 2024On Barycenter Computation: Semi-Unbalanced Optimal Transport-based Method on Gaussianscited by 0
  6. 2026MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SCALPEL: Semantic Cross-modal Alignment via LLM-Powered Encoder Learning for Medical Vision-Language Representation
  2. 2026MedLayBench-V: A Large-Scale Benchmark for Expert-Lay Semantic Alignment in Medical Vision Language Models
  3. 2026BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation
  4. 2026LLM-Guided Diagnostic Evidence Alignment for Medical Vision-Language Pretraining under Limited Pairing
  5. 2026Can LLM-Generated Text Empower Surgical Vision-Language Pre-training?
  6. 2026Towards Clinically Faithful Medical Image Captioning via Enhanced Vision-Language Alignment
  7. 2025Enhancing Medical Large Vision-Language Models via Alignment Distillation
  8. 2025More performant and scalable: Rethinking contrastive vision-language pre-training of radiology in the LLM era