# Gaze-Assisted Medical Image Segmentation

arXiv: https://arxiv.org/abs/2410.17920 · HTML: https://mlsanity.com/paper/2410.17920/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 35 references that are in the map.
- First posted on arXiv 23 Oct 2024.

## Builds on (most cited first, 10 of 10)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data](https://mlsanity.com/paper/1904.00592/) (2019)
- [A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1910.02923/) (2019)
- [Creation and Validation of a Chest X-Ray Dataset with Eye-tracking and Report Dictation for AI Development](https://mlsanity.com/paper/2009.07386/) (2020)
- [Follow My Eye: Using Gaze to Supervise Computer-Aided Diagnosis](https://mlsanity.com/paper/2204.02976/) (2022)
- [Explainable Medical Imaging AI Needs Human-Centered Design: Guidelines and Evidence from a Systematic Review](https://mlsanity.com/paper/2112.12596/) (2021)
- [Segment Anything Model (SAM) for Radiation Oncology](https://mlsanity.com/paper/2306.11730/) (2023)
- [Gaze2Segment: A Pilot Study for Integrating Eye-Tracking Technology into Medical Image Segmentation](https://mlsanity.com/paper/1608.03235/) (2016)

## Built on by (most cited first, 4 of 4)

- [Zero-Shot Gaze-based Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2505.15256/) (2025)
- [Eye on the Target: Eye Tracking Meets Rodent Tracking](https://mlsanity.com/paper/2503.10305/) (2025)
- [Optimization of MedSAM model based on bounding box adaptive perturbation algorithm](https://mlsanity.com/paper/2503.19700/) (2025)
- [GazeXPErT: An Expert Eye-tracking Dataset for Interpretable and Explainable AI in Oncologic FDG-PET/CT Scans](https://mlsanity.com/paper/2603.00162/) (2026)

## Abstract (arXiv)

The annotation of patient organs is a crucial part of various diagnostic and treatment procedures, such as radiotherapy planning. Manual annotation is extremely time-consuming, while its automation using modern image analysis techniques has not yet reached levels sufficient for clinical adoption. This paper investigates the idea of semi-supervised medical image segmentation using human gaze as interactive input for segmentation correction. In particular, we fine-tuned the Segment Anything Model in Medical Images (MedSAM), a public solution that uses various prompt types as additional input for semi-automated segmentation correction. We used human gaze data from reading abdominal images as a prompt for fine-tuning MedSAM. The model was validated on a public WORD database, which consists of 120 CT scans of 16 abdominal organs. The results of the gaze-assisted MedSAM were shown to be superior to the results of the state-of-the-art segmentation models. In particular, the average Dice coefficient for 16 abdominal organs was 85.8%, 86.7%, 81.7%, and 90.5% for nnUNetV2, ResUNet, original MedSAM, and our gaze-assisted MedSAM model, respectively.