mlsanityData to 7 Oct 2026

Zero-Shot Segmentation of Eye Features Using the Segment Anything Model (SAM)

arXiv 2311.08077 ↗First posted 14 Nov 2023cs.CV cs.AI cs.HCcited by 7

Not on the map. Why?

Abstract (arXiv)

The advent of foundation models signals a new era in artificial intelligence. The Segment Anything Model (SAM) is the first foundation model for image segmentation. In this study, we evaluate SAM's ability to segment features from eye images recorded in virtual reality setups. The increasing requirement for annotated eye-image datasets presents a significant opportunity for SAM to redefine the landscape of data annotation in gaze estimation. Our investigation centers on SAM's zero-shot learning abilities and the effectiveness of prompts like bounding boxes or point clicks. Our results are consistent with studies in other domains, demonstrating that SAM's segmentation effectiveness can be on-par with specialized models depending on the feature, with prompts improving its performance, evidenced by an IoU of 93.34% for pupil segmentation in one dataset. Foundation models like SAM could revolutionize gaze estimation by enabling quick and easy image segmentation, reducing reliance on specialized models and extensive manual annotation.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 27 of its 48 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

27 of 48 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023Segment Anythingcited by 8,158
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
  8. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,483
  9. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 364
  10. 2023Segment Anything Model for Medical Image Analysis: an Experimental Studycited by 291
  11. 2023Track Anything: Segment Anything Meets Videoscited by 230
  12. 2023Segment Anything in Medical Imagescited by 226
15 more
  1. 2019IoU Loss for 2D/3D Object Detectioncited by 128
  2. 2023Anything-3D: Towards Single-view Anything Reconstruction in the Wildcited by 91
  3. 2023SAMRS: Scaling-up Remote Sensing Segmentation Dataset with Segment Anything Modelcited by 38
  4. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labellingcited by 37
  5. 2021TEyeD: Over 20 million real-world eye images with Pupil, Eyelid, and Iris 2D and 3D Segmentations, 2D and 3D Landmarks, 3D Eyeball, Gaze Vector, and Eye Movement Typescited by 32
  6. 2019Gaze-in-wild: A dataset for studying eye and head coordination in everyday activitiescited by 25
  7. 2020EllSeg: An Ellipse Segmentation Framework for Robust Gaze Trackingcited by 24
  8. 2016PupilNet: Convolutional Neural Networks for Robust Pupil Detectioncited by 20
  9. 2023GazeSAM: What You See is What You Segmentcited by 19
  10. 2017PuRe: Robust pupil detection for real-time pervasive eye trackingcited by 17
  11. 2020RIT-Eyes: Rendering of near-eye images for eye-tracking applicationscited by 10
  12. 2022Pistol: Pupil Invisible Supportive Tool to extract Pupil, Iris, Eye Opening, Eye Movements, Pupil and Iris Gaze Vector, and 2D as well as 3D Gazecited by 9
  13. 2022EllSeg-Gen, towards Domain Generalization for head-mounted eyetrackingcited by 6
  14. 2023Precise localization of corneal reflections in eye images using deep learning trained on synthetic datacited by 4
  15. 2023LEyes: A Lightweight Framework for Deep Learning-Based Eye Tracking using Synthetic Eye Imagescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 0; 2026: 1.

7 papers.

  1. 2024Zero-Shot Scene Change Detectioncited by 13
  2. 2024Zero-Shot Automatic Annotation and Instance Segmentation using LLM-Generated Datasets: Eliminating Field Imaging and Manual Annotation for Deep Learning Model Developmentcited by 5
  3. 2024Auto-Generating Weak Labels for Real & Synthetic Data to Improve Label-Scarce Medical Image Segmentationcited by 3
  4. 2024Towards Unsupervised Eye-Region Segmentation for Eye Trackingcited by 2
  5. 2024Zero-Shot Pupil Segmentation with SAM 2: A Case Study of Over 14 Million Imagescited by 2
  6. 2024VideoSAM: A Large Vision Foundation Model for High-Speed Video Segmentationcited by 1
  7. 2026Eye image segmentation using visual and concept prompts with Segment Anything Model 3 (SAM3)cited by 0