# Meta-Entity Driven Triplet Mining for Aligning Medical Vision-Language Models

arXiv: https://arxiv.org/abs/2504.15929 · HTML: https://mlsanity.com/paper/2504.15929/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 69 references that are in the map.
- First posted on arXiv 22 Apr 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 18 of 18)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretation](https://mlsanity.com/paper/2401.12208/) (2024)
- [Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias](https://mlsanity.com/paper/2305.19894/) (2023)
- [Detecting and Evaluating Medical Hallucinations in Large Vision Language Models](https://mlsanity.com/paper/2406.10185/) (2024)
- [Self-Consistent Recursive Diffusion Bridge for Medical Image Translation](https://mlsanity.com/paper/2405.06789/) (2024)
- [I2I-Mamba: Multi-modal medical image synthesis via selective state space modeling](https://mlsanity.com/paper/2405.14022/) (2024)
- [VSE++: Improving Visual-Semantic Embeddings with Hard Negatives](https://mlsanity.com/paper/1707.05612/) (2017)
- [Anatomical Structure-Guided Medical Vision-Language Pre-training](https://mlsanity.com/paper/2403.09294/) (2024)
- [Self-supervised Image-text Pre-training With Mixed Data In Chest X-rays](https://mlsanity.com/paper/2103.16022/) (2021)
- [Utilizing Synthetic Data for Medical Vision-Language Pre-training: Bypassing the Need for Real Images](https://mlsanity.com/paper/2310.07027/) (2023)
- [X-TRA: Improving Chest X-ray Tasks with Cross-Modal Retrieval Augmentation](https://mlsanity.com/paper/2302.11352/) (2023)
- [Physics-Driven Autoregressive State Space Models for Medical Image Reconstruction](https://mlsanity.com/paper/2412.09331/) (2024)
- [MLIP: Medical Language-Image Pre-training with Masked Local Representation Learning](https://mlsanity.com/paper/2401.01591/) (2024)
- [Generative Autoregressive Transformers for Model-Agnostic Federated MRI Reconstruction](https://mlsanity.com/paper/2502.04521/) (2025)
- [Enhancing medical vision-language contrastive learning via inter-matching relation modelling](https://mlsanity.com/paper/2401.10501/) (2024)
- [Enhancing Representation in Medical Vision-Language Foundation Models via Multi-Scale Information Extraction Techniques](https://mlsanity.com/paper/2401.01583/) (2024)

## Built on by (most cited first, 1 of 1)

- [ACE-LoRA: Graph-Attentive Context Enhancement for Parameter-Efficient Adaptation of Medical Vision-Language Models](https://mlsanity.com/paper/2603.17079/) (2026)

## Abstract (arXiv)

Diagnostic imaging relies on interpreting both images and radiology reports, but the growing data volumes place significant pressure on medical experts, yielding increased errors and workflow backlogs. Medical vision-language models (med-VLMs) have emerged as a powerful framework to efficiently process multimodal imaging data, particularly in chest X-ray (CXR) evaluations, albeit their performance hinges on how well image and text representations are aligned. Existing alignment methods, predominantly based on contrastive learning, prioritize separation between disease classes over segregation of fine-grained pathology attributes like location, size or severity, leading to suboptimal representations. Here, we propose MedTrim (Meta-entity-driven Triplet mining), a novel method that enhances image-text alignment through multimodal triplet learning synergistically guided by disease class as well as adjectival and directional pathology descriptors. Unlike common alignment methods that separate broad disease classes, MedTrim leverages structured meta-entity information to preserve subtle but clinically significant intra-class variations. For this purpose, we first introduce an ontology-based entity recognition module that extracts pathology-specific meta-entities from CXR reports, as annotations on pathology attributes are rare in public datasets. For refined sample selection in triplet mining, we then introduce a novel score function that captures an aggregate measure of inter-sample similarity based on disease classes and adjectival/directional descriptors. Lastly, we introduce a multimodal triplet alignment objective for explicit within- and cross-modal alignment between samples sharing detailed pathology characteristics. Our demonstrations indicate that MedTrim improves performance in downstream retrieval and classification tasks compared to state-of-the-art alignment methods.