# Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images

arXiv: https://arxiv.org/abs/2508.00549 · HTML: https://mlsanity.com/paper/2508.00549/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 16 of its 37 references that are in the map.
- First posted on arXiv 1 Aug 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 16 of 16)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model](https://mlsanity.com/paper/2312.11370/) (2023)
- [Pixtral 12B](https://mlsanity.com/paper/2410.07073/) (2024)
- [ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts](https://mlsanity.com/paper/2312.00784/) (2023)
- [Vision language models are blind: Failing to translate detailed visual features into words](https://mlsanity.com/paper/2407.06581/) (2024)
- [Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review](https://mlsanity.com/paper/2403.02469/) (2024)
- [Vision Language Models are Biased](https://mlsanity.com/paper/2505.23941/) (2025)
- [Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective](https://mlsanity.com/paper/2403.18346/) (2024)
- [Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine](https://mlsanity.com/paper/2401.08396/) (2024)
- [Understanding Spatial Language in Radiology: Representation Framework, Annotation, and Spatial Relation Extraction from Chest X-ray Reports using Deep Learning](https://mlsanity.com/paper/1908.04485/) (2019)
- [Beyond the Hype: A dispassionate look at vision-language models in medical scenario](https://mlsanity.com/paper/2408.08704/) (2024)

## Built on by (most cited first, 13 of 13)

- [Lost in Volume: The CT-SpatialVQA Benchmark for Evaluating Semantic-Spatial Understanding of 3D Medical Vision-Language Models](https://mlsanity.com/paper/2605.08787/) (2026)
- [AnatomiX, an Anatomy-Aware Grounded Multimodal Large Language Model for Chest X-Ray Interpretation](https://mlsanity.com/paper/2601.03191/) (2026)
- [PaTAS: A Framework for Trust Propagation in Neural Networks Using Subjective Logic](https://mlsanity.com/paper/2511.20586/) (2025)
- [Medical Image Spatial Grounding with Semantic Sampling](https://mlsanity.com/paper/2603.14579/) (2026)
- [ExpressEdit: Fast Editing of Stylized Facial Expressions with Diffusion Models in Photoshop](https://mlsanity.com/paper/2604.03448/) (2026)
- [Follow the Mean: Reference-Guided Flow Matching](https://mlsanity.com/paper/2605.10302/) (2026)
- [DS@GT ARC at ImageCLEFmedical 2026: Architectural Diversity for Concept Detection and Foundation-Model Scaling for Caption Prediction in Medical Image Analysis](https://mlsanity.com/paper/2607.27763/) (2026)
- [Open-Linguistic Concept Unified Learning for Cross-Site Interpretable Dermatology Image Diagnosis](https://mlsanity.com/paper/2608.03225/) (2026)
- [CoM3eT: A foundation model for medical image analysis through federated, multidimensional context integration](https://mlsanity.com/paper/2608.16268/) (2026)
- [A Modular Agent for Reliable and Auditable Spatial Relation Verification in CT Scans](https://mlsanity.com/paper/2608.21140/) (2026)
- [AgenTeeth: A Model-Agnostic Framework for Suppressing Hallucination in Frozen Vision-Language Models on Dental X-Rays via Tool Evidence Injection](https://mlsanity.com/paper/2609.17800/) (2026)
- [Who Is Left of Whom? Tracing Spatial Evidence and Role Binding in Relative-Position Reasoning](https://mlsanity.com/paper/2609.35486/) (2026)
- [Can machines reason non-verbally?](https://mlsanity.com/paper/2610.13925/) (2026)

## Abstract (arXiv)

Clinical decision-making relies heavily on understanding relative positions of anatomical structures and anomalies. Therefore, for Vision-Language Models (VLMs) to be applicable in clinical practice, the ability to accurately determine relative positions on medical images is a fundamental prerequisite. Despite its importance, this capability remains highly underexplored. To address this gap, we evaluate the ability of state-of-the-art VLMs, GPT-4o, Llama3.2, Pixtral, and JanusPro, and find that all models fail at this fundamental task. Inspired by successful approaches in computer vision, we investigate whether visual prompts, such as alphanumeric or colored markers placed on anatomical structures, can enhance performance. While these markers provide moderate improvements, results remain significantly lower on medical images compared to observations made on natural images. Our evaluations suggest that, in medical imaging, VLMs rely more on prior anatomical knowledge than on actual image content for answering relative position questions, often leading to incorrect conclusions. To facilitate further research in this area, we introduce the MIRP , Medical Imaging Relative Positioning, benchmark dataset, designed to systematically evaluate the capability to identify relative positions in medical images.