# A Modular Agent for Reliable and Auditable Spatial Relation Verification in CT Scans

arXiv: https://arxiv.org/abs/2608.21140 · HTML: https://mlsanity.com/paper/2608.21140/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 21 Aug 2026.

## Builds on (most cited first, 18 of 18)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [MedGemma Technical Report](https://mlsanity.com/paper/2507.05201/) (2025)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [What's "up" with vision-language models? Investigating their struggle with spatial reasoning](https://mlsanity.com/paper/2310.19785/) (2023)
- [Pixtral 12B](https://mlsanity.com/paper/2410.07073/) (2024)
- [MMedAgent: Learning to Use Medical Tools with Multi-modal Agent](https://mlsanity.com/paper/2407.02483/) (2024)
- [MedRAX: Medical Reasoning Agent for Chest X-ray](https://mlsanity.com/paper/2502.02673/) (2025)
- [VividMed: Vision Language Model with Versatile Visual Grounding for Medicine](https://mlsanity.com/paper/2410.12694/) (2024)
- [Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images](https://mlsanity.com/paper/2508.00549/) (2025)
- [Understanding Spatial Language in Radiology: Representation Framework, Annotation, and Spatial Relation Extraction from Chest X-ray Reports using Deep Learning](https://mlsanity.com/paper/1908.04485/) (2019)
- [A Multimodal Multi-Agent Framework for Radiology Report Generation](https://mlsanity.com/paper/2505.09787/) (2025)
- [VoxelPrompt: A Vision Agent for End-to-End Medical Image Analysis](https://mlsanity.com/paper/2410.08397/) (2024)
- [RadAgent: A tool-using AI agent for stepwise interpretation of chest computed tomography](https://mlsanity.com/paper/2604.15231/) (2026)
- [Agentic Large Language Models for Training-Free Neuro-Radiological Image Analysis](https://mlsanity.com/paper/2604.16729/) (2026)

## Abstract (arXiv)

Reliable spatial understanding is an important prerequisite for future medical vision-language systems that aim to support radiological report generation and structured image understanding. While modern vision-language models (VLMs) show promising performance on many medical imaging tasks, recent evidence suggests they remain weak in controlled spatial reasoning and often fail to reliably ground spatial relations in image evidence. Given that radiological reasoning hinges on understanding the relative positions of anatomical structures and findings, this spatial weakness poses risks to diagnostic accuracy. We present a modular medical imaging agent for binary spatial relation verification in axial CT slices. Instead of directly predicting spatial answers end-to-end, the system decomposes the task into explicit stages: language parsing, anatomical localization, and deterministic geometric verification. Natural-language queries are converted into structured relation tuples, queried organs are localized with a YOLO-based detector, and the final spatial decision is computed from object centers using deterministic geometric rules. We evaluate the approach on the held-out MIRP spatial QA benchmark and compare it against representative end-to-end VLM baselines. The best-performing hybrid configuration reaches 94.1% accuracy and 94.2% F1, outperforming direct Qwen2-VL prompting by 42.5 percentage points in accuracy, while preserving interpretable intermediate representations and auditable reasoning stages. The results suggest that explicit modular spatial verification can serve as a promising building block for future report-oriented medical imaging agents.