# AmbientEye: A Dataset for Pupil Segmentation under Natural Ambient Infrared Illumination

arXiv: https://arxiv.org/abs/2606.03774 · HTML: https://mlsanity.com/paper/2606.03774/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 39 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Builds on (most cited first, 16 of 16)

- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Project Aria: A New Tool for Egocentric Multi-Modal AI Research](https://mlsanity.com/paper/2308.13561/) (2023)
- [RITnet: Real-time Semantic Segmentation of the Eye for Gaze Tracking](https://mlsanity.com/paper/1910.00694/) (2019)
- [TEyeD: Over 20 million real-world eye images with Pupil, Eyelid, and Iris 2D and 3D Segmentations, 2D and 3D Landmarks, 3D Eyeball, Gaze Vector, and Eye Movement Types](https://mlsanity.com/paper/2102.02115/) (2021)
- [EllSeg: An Ellipse Segmentation Framework for Robust Gaze Tracking](https://mlsanity.com/paper/2007.09600/) (2020)
- [Labeled pupils in the wild: A dataset for studying pupil detection in unconstrained environments](https://mlsanity.com/paper/1511.05768/) (2015)
- [PuRe: Robust pupil detection for real-time pervasive eye tracking](https://mlsanity.com/paper/1712.08900/) (2017)
- [GazeGPT: Augmenting Human Capabilities using Gaze-contingent Contextual AI for Smart Eyewear](https://mlsanity.com/paper/2401.17217/) (2024)
- [PupilNet v2.0: Convolutional Neural Networks for CPU based real time Robust Pupil Detection](https://mlsanity.com/paper/1711.00112/) (2017)
- [A High-Level Description and Performance Evaluation of Pupil Invisible](https://mlsanity.com/paper/2009.00508/) (2020)
- [Aria Gen 2 Pilot Dataset](https://mlsanity.com/paper/2510.16134/) (2025)
- [MagicEyes: A Large Scale Eye Gaze Estimation Dataset for Mixed Reality](https://mlsanity.com/paper/2003.08806/) (2020)
- [Rethinking Eye-blink: Assessing Task Difficulty through Physiological Representation of Spontaneous Blinking](https://mlsanity.com/paper/2102.06690/) (2021)
- [GazeTrak: Exploring Acoustic-based Eye Tracking on a Glass Frame](https://mlsanity.com/paper/2402.14634/) (2024)
- [Zero-Shot Pupil Segmentation with SAM 2: A Case Study of Over 14 Million Images](https://mlsanity.com/paper/2410.08926/) (2024)

## Abstract (arXiv)

Eye tracking is essential for smart glasses, as it provides insight into user attention for ambient intelligence applications. However, most existing eye-tracking systems rely on active infrared (IR) illumination, creating practical barriers to all-day outdoor use due to power consumption. In this paper, we investigate whether passive IR cameras alone, without any active IR light source, can enable reliable pupil detection in unconstrained outdoor environments, where ambient sunlight serves as the sole illumination source. To support this investigation, we introduce AmbientEye, a large-scale dataset of 2,606,225 eye images collected from 35 participants from 19 countries. It is captured outdoors under natural sunlight with two off-axis camera configurations and two sun-orientation conditions. We provide high-quality pupil annotation through SAM2 automatic segmentation, followed by refinement by human annotators. We benchmark a state-of-the-art pupil segmentation algorithm on our dataset and compare its performance with that on existing datasets under controlled IR illumination. Results reveal a substantial drop in pupil segmentation performance from 0.928 on controlled IR datasets to 0.767 on AmbientEye. This performance gap highlights the challenge of the ambient-light setting. This positions AmbientEye as a first benchmark for an unexplored and highly practical eye-tracking scenario.