# Weakly-supervised Medical Image Segmentation with Gaze Annotations

arXiv: https://arxiv.org/abs/2407.07406 · HTML: https://mlsanity.com/paper/2407.07406/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 10 Jul 2024.

## Builds on (most cited first, 17 of 17)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [Early-Learning Regularization Prevents Memorization of Noisy Labels](https://mlsanity.com/paper/2007.00151/) (2020)
- [Unsupervised Semantic Segmentation by Distilling Feature Correspondences](https://mlsanity.com/paper/2203.08414/) (2022)
- [BoxInst: High-Performance Instance Segmentation with Box Annotations](https://mlsanity.com/paper/2012.02310/) (2020)
- [ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models](https://mlsanity.com/paper/2302.07257/) (2023)
- [Pointly-Supervised Instance Segmentation](https://mlsanity.com/paper/2104.06404/) (2021)
- [Extreme clicking for efficient object annotation](https://mlsanity.com/paper/1708.02750/) (2017)
- [Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization](https://mlsanity.com/paper/2202.10341/) (2022)
- [Follow My Eye: Using Gaze to Supervise Computer-Aided Diagnosis](https://mlsanity.com/paper/2204.02976/) (2022)
- [BoxTeacher: Exploring High-Quality Pseudo Labels for Weakly Supervised Instance Segmentation](https://mlsanity.com/paper/2210.05174/) (2022)
- [Mitigating Memorization of Noisy Labels via Regularization between Representations](https://mlsanity.com/paper/2110.09022/) (2021)
- [Mixup-CAM: Weakly-supervised Semantic Segmentation via Uncertainty Regularization](https://mlsanity.com/paper/2008.01201/) (2020)
- [Leveraging Human Selective Attention for Medical Image Analysis with Limited Training Data](https://mlsanity.com/paper/2112.01034/) (2021)

## Built on by (most cited first, 10 of 10)

- [From Gaze to Insight: Bridging Human Visual Attention and Vision Language Model Explanation for Weakly-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2504.11368/) (2025)
- [Enjoying Information Dividend: Gaze Track-based Medical Weakly Supervised Segmentation](https://mlsanity.com/paper/2505.22230/) (2025)
- [Improvise, Adapt, Overcome -- Telescopic Adapters for Efficient Fine-tuning of Vision Language Models in Medical Imaging](https://mlsanity.com/paper/2512.13855/) (2025)
- [Eye on the Target: Eye Tracking Meets Rodent Tracking](https://mlsanity.com/paper/2503.10305/) (2025)
- [TWLR: Text-Guided Weakly-Supervised Lesion Localization and Severity Regression for Explainable Diabetic Retinopathy Grading](https://mlsanity.com/paper/2512.13008/) (2025)
- [OBBSeg: Irregular Lesion Segmentation under Oriented Bounding Box Annotations](https://mlsanity.com/paper/2607.06007/) (2026)
- [From Spatial Semantics to Temporal Context: Leveraging Gaze Trajectory for Weakly Supervised Medical Image Segmentation](https://mlsanity.com/paper/2607.26542/) (2026)
- [Measuring Browser Webcam Gaze Honestly: A Capture-Clock Methodology and Open Reference Implementation](https://mlsanity.com/paper/2608.11566/) (2026)
- [SGRNet: Spatially Guided Radiology Network for Structured Radiological Reporting of Head and Neck Cancer](https://mlsanity.com/paper/2608.29153/) (2026)
- [GazeRefine: Expert Gaze as a Test-Time Prompt for Training-Free Medical Image Segmentation](https://mlsanity.com/paper/2609.01310/) (2026)

## Abstract (arXiv)

Eye gaze that reveals human observational patterns has increasingly been incorporated into solutions for vision tasks. Despite recent explorations on leveraging gaze to aid deep networks, few studies exploit gaze as an efficient annotation approach for medical image segmentation which typically entails heavy annotating costs. In this paper, we propose to collect dense weak supervision for medical image segmentation with a gaze annotation scheme. To train with gaze, we propose a multi-level framework that trains multiple networks from discriminative human attention, simulated with a set of pseudo-masks derived by applying hierarchical thresholds on gaze heatmaps. Furthermore, to mitigate gaze noise, a cross-level consistency is exploited to regularize overfitting noisy labels, steering models toward clean patterns learned by peer networks. The proposed method is validated on two public medical datasets of polyp and prostate segmentation tasks. We contribute a high-quality gaze dataset entitled GazeMedSeg as an extension to the popular medical segmentation datasets. To the best of our knowledge, this is the first gaze dataset for medical image segmentation. Our experiments demonstrate that gaze annotation outperforms previous label-efficient annotation schemes in terms of both performance and annotation time. Our collected gaze data and code are available at: https://github.com/med-air/GazeMedSeg.