# GazeRefine: Expert Gaze as a Test-Time Prompt for Training-Free Medical Image Segmentation

arXiv: https://arxiv.org/abs/2609.01310 · HTML: https://mlsanity.com/paper/2609.01310/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Builds on (most cited first, 24 of 24)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [What's the Point: Semantic Segmentation with Point Supervision](https://mlsanity.com/paper/1506.02106/) (2015)
- [Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation](https://mlsanity.com/paper/1908.10454/) (2019)
- [Personalize Segment Anything Model with One Shot](https://mlsanity.com/paper/2305.03048/) (2023)
- [Medical Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2009.13120/) (2020)
- [BoxInst: High-Performance Instance Segmentation with Box Annotations](https://mlsanity.com/paper/2012.02310/) (2020)
- [Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching](https://mlsanity.com/paper/2305.13310/) (2023)
- [Test-Time Adaptable Neural Networks for Robust Medical Image Segmentation](https://mlsanity.com/paper/2004.04668/) (2020)
- [Scribble-Supervised Medical Image Segmentation via Dual-Branch Network and Dynamically Mixed Pseudo Labels Supervision](https://mlsanity.com/paper/2203.02106/) (2022)
- [NiftyNet: a deep-learning platform for medical imaging](https://mlsanity.com/paper/1709.03485/) (2017)
- [Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2411.19331/) (2024)
- [BoxTeacher: Exploring High-Quality Pseudo Labels for Weakly Supervised Instance Segmentation](https://mlsanity.com/paper/2210.05174/) (2022)
- [GazeSAM: What You See is What You Segment](https://mlsanity.com/paper/2304.13844/) (2023)
- [Gaze2Segment: A Pilot Study for Integrating Eye-Tracking Technology into Medical Image Segmentation](https://mlsanity.com/paper/1608.03235/) (2016)
- [Weakly-supervised Medical Image Segmentation with Gaze Annotations](https://mlsanity.com/paper/2407.07406/) (2024)
- [Prompt Engineering in Segment Anything Model: Methodologies, Applications, and Emerging Challenges](https://mlsanity.com/paper/2507.09562/) (2025)
- [Zero-Shot Gaze-based Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2505.15256/) (2025)
- [Enjoying Information Dividend: Gaze Track-based Medical Weakly Supervised Segmentation](https://mlsanity.com/paper/2505.22230/) (2025)

## Abstract (arXiv)

Medical image segmentation remains difficult to scale because high-performing methods typically rely on dense expert annotations and task-specific training. We introduce GazeRefine, a training-free framework that uses gaze as an inference-time prompt for zero-shot medical image segmentation. Sparse, duration-weighted fixations are converted into foreground and background priors that initialize semantic prototypes in frozen DINOv3 feature space. These prototypes are iteratively refined through foreground-background discrimination, feature-space affinity propagation, and anchoring to the initial gaze guidance, allowing segmentation to extend beyond directly fixated regions while limiting semantic drift. GazeRefine requires no segmentation masks, fine-tuning, adapters, prompt encoders, or gradient updates. We evaluate the method on gaze-annotated polyp segmentation and prostate MRI segmentation. The results show strong performance on colonoscopy images and competitive performance on prostate MRI, supporting gaze-guided prototype refinement as a promising approach for segmentation-label-efficient, human-in-the-loop medical image segmentation. Our tools and code can be found in the following repository: https://github.com/MohammedOussamaBEN/GazeRefine.git