# Zero-Shot Pupil Segmentation with SAM 2: A Case Study of Over 14 Million Images

arXiv: https://arxiv.org/abs/2410.08926 · HTML: https://mlsanity.com/paper/2410.08926/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 71 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Builds on (most cited first, 30 of 44)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [Fast Segment Anything](https://mlsanity.com/paper/2306.12156/) (2023)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [Inpaint Anything: Segment Anything Meets Image Inpainting](https://mlsanity.com/paper/2304.06790/) (2023)
- [Segment Anything in Medical Images](https://mlsanity.com/paper/2304.12306/) (2023)
- [Segment and Track Anything](https://mlsanity.com/paper/2305.06558/) (2023)
- [TimeGPT-1](https://mlsanity.com/paper/2310.03589/) (2023)
- [SAM3D: Segment Anything in 3D Scenes](https://mlsanity.com/paper/2306.03908/) (2023)
- [Medical SAM 2: Segment medical images as video via Segment Anything Model 2](https://mlsanity.com/paper/2408.00874/) (2024)
- [On the Challenges and Perspectives of Foundation Models for Medical Image Analysis](https://mlsanity.com/paper/2306.05705/) (2023)
- [Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directions](https://mlsanity.com/paper/2401.03495/) (2024)
- [Anything-3D: Towards Single-view Anything Reconstruction in the Wild](https://mlsanity.com/paper/2304.10261/) (2023)
- [On the Societal Impact of Open Foundation Models](https://mlsanity.com/paper/2403.07918/) (2024)
- [Adapting Segment Anything Model for Change Detection in HR Remote Sensing Images](https://mlsanity.com/paper/2309.01429/) (2023)
- [MobileSAMv2: Faster Segment Anything to Everything](https://mlsanity.com/paper/2312.09579/) (2023)
- [TEyeD: Over 20 million real-world eye images with Pupil, Eyelid, and Iris 2D and 3D Segmentations, 2D and 3D Landmarks, 3D Eyeball, Gaze Vector, and Eye Movement Types](https://mlsanity.com/paper/2102.02115/) (2021)
- [OpenEDS: Open Eye Dataset](https://mlsanity.com/paper/1905.03702/) (2019)
- [DinoBloom: A Foundation Model for Generalizable Cell Embeddings in Hematology](https://mlsanity.com/paper/2404.05022/) (2024)
- [Gaze-in-wild: A dataset for studying eye and head coordination in everyday activities](https://mlsanity.com/paper/1905.13146/) (2019)
- [PupilNet: Convolutional Neural Networks for Robust Pupil Detection](https://mlsanity.com/paper/1601.04902/) (2016)
- [OpenEDS2020: Open Eyes Dataset](https://mlsanity.com/paper/2005.03876/) (2020)
- [Labeled pupils in the wild: A dataset for studying pupil detection in unconstrained environments](https://mlsanity.com/paper/1511.05768/) (2015)

## Built on by (most cited first, 2 of 2)

- [Eye image segmentation using visual and concept prompts with Segment Anything Model 3 (SAM3)](https://mlsanity.com/paper/2603.17715/) (2026)
- [AmbientEye: A Dataset for Pupil Segmentation under Natural Ambient Infrared Illumination](https://mlsanity.com/paper/2606.03774/) (2026)

## Abstract (arXiv)

We explore the transformative potential of SAM 2, a vision foundation model, in advancing gaze estimation and eye tracking technologies. By significantly reducing annotation time, lowering technical barriers through its ease of deployment, and enhancing segmentation accuracy, SAM 2 addresses critical challenges faced by researchers and practitioners. Utilizing its zero-shot segmentation capabilities with minimal user input-a single click per video-we tested SAM 2 on over 14 million eye images from diverse datasets, including virtual reality setups and the world's largest unified dataset recorded using wearable eye trackers. Remarkably, in pupil segmentation tasks, SAM 2 matches the performance of domain-specific models trained solely on eye images, achieving competitive mean Intersection over Union (mIoU) scores of up to 93% without fine-tuning. Additionally, we provide our code and segmentation masks for these widely used datasets to promote further research.