# Towards user-centered interactive medical image segmentation in VR with an assistive AI agent

arXiv: https://arxiv.org/abs/2505.07214 · HTML: https://mlsanity.com/paper/2505.07214/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 43 references that are in the map.
- First posted on arXiv 12 May 2025.

## Builds on (most cited first, 14 of 14)

- [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019)
- [The Faiss library](https://mlsanity.com/paper/2401.08281/) (2024)
- [Medical SAM 2: Segment medical images as video via Segment Anything Model 2](https://mlsanity.com/paper/2408.00874/) (2024)
- [A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1910.02923/) (2019)
- [BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at once](https://mlsanity.com/paper/2405.12971/) (2024)
- [Interactive 3D Medical Image Segmentation with SAM 2](https://mlsanity.com/paper/2408.02635/) (2024)
- [VR-GPT: Visual Language Model for Intelligent Virtual Reality Applications](https://mlsanity.com/paper/2405.11537/) (2024)
- [Transcending Dimensions using Generative AI: Real-Time 3D Model Generation in Augmented Reality](https://mlsanity.com/paper/2504.21033/) (2025)
- [Towards Anatomy Education with Generative AI-based Virtual Assistants in Immersive Virtual Reality Environments](https://mlsanity.com/paper/2306.17278/) (2023)
- [A general-purpose AI assistant embedded in an open-source radiology information system](https://mlsanity.com/paper/2303.10338/) (2023)
- [From Voices to Worlds: Developing an AI-Powered Framework for 3D Object Generation in Augmented Reality](https://mlsanity.com/paper/2503.16474/) (2025)
- [Evaluation of Text Selection Techniques in Virtual Reality Head-Mounted Displays](https://mlsanity.com/paper/2209.06498/) (2022)
- [Assessment of user-interaction strategies for neurosurgical data navigation and annotation in virtual reality](https://mlsanity.com/paper/2212.08688/) (2022)
- [SONIA: an immersive customizable virtual reality system for the education and exploration of brain networks](https://mlsanity.com/paper/2301.09772/) (2023)

## Abstract (arXiv)

Crucial in disease analysis and surgical planning, manual segmentation of volumetric medical scans (e.g. MRI, CT) is laborious, error-prone, and challenging to master, while fully automatic algorithms can benefit from user feedback. Therefore, with the complementary power of the latest radiological AI foundation models and virtual reality (VR)'s intuitive data interaction, we propose SAMIRA, a novel conversational AI agent for medical VR that assists users with localizing, segmenting, and visualizing 3D medical concepts. Through speech-based interaction, the agent helps users understand radiological features, locate clinical targets, and generate segmentation masks that can be refined with just a few point prompts. The system also supports true-to-scale 3D visualization of segmented pathology to enhance patient-specific anatomical understanding. Furthermore, to determine the optimal interaction paradigm under near-far attention-switching for refining segmentation masks in an immersive, human-in-the-loop workflow, we compare VR controller pointing, head pointing, and eye tracking as input modes. With a user study, evaluations demonstrated a high usability score (SUS=90.0 ± 9.0), low overall task load, as well as strong support for the proposed VR system's guidance, training potential, and integration of AI in radiological segmentation tasks.