# Gaze-in-wild: A dataset for studying eye and head coordination in everyday activities

arXiv: https://arxiv.org/abs/1905.13146 · HTML: https://mlsanity.com/paper/1905.13146/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 May 2019.

## Built on by (most cited first, 25 of 25)

- [BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments](https://mlsanity.com/paper/2108.03332/) (2021)
- [MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain](https://mlsanity.com/paper/2209.08691/) (2022)
- [Automatic Gaze Analysis: A Survey of Deep Learning based Approaches](https://mlsanity.com/paper/2108.05479/) (2021)
- [TEyeD: Over 20 million real-world eye images with Pupil, Eyelid, and Iris 2D and 3D Segmentations, 2D and 3D Landmarks, 3D Eyeball, Gaze Vector, and Eye Movement Types](https://mlsanity.com/paper/2102.02115/) (2021)
- [An End-to-End Review of Gaze Estimation and its Interactive Applications on Handheld Mobile Devices](https://mlsanity.com/paper/2307.00122/) (2023)
- [GazeMoDiff: Gaze-guided Diffusion Model for Stochastic Human Motion Prediction](https://mlsanity.com/paper/2312.12090/) (2023)
- [A Multimodal Eye Movement Dataset and a Multimodal Eye Movement Segmentation Analysis](https://mlsanity.com/paper/2101.04318/) (2021)
- [Zero-Shot Segmentation of Eye Features Using the Segment Anything Model (SAM)](https://mlsanity.com/paper/2311.08077/) (2023)
- [Pose2Gaze: Eye-body Coordination during Daily Activities for Gaze Prediction from Full-body Poses](https://mlsanity.com/paper/2312.12042/) (2023)
- [Few-shot Geometry-Aware Keypoint Localization](https://mlsanity.com/paper/2303.17216/) (2023)
- [LEyes: A Lightweight Framework for Deep Learning-Based Eye Tracking using Synthetic Eye Images](https://mlsanity.com/paper/2309.06129/) (2023)
- [The Visual Experience Dataset: Over 200 Recorded Hours of Integrated Eye Movement, Odometry, and Egocentric Video](https://mlsanity.com/paper/2404.18934/) (2024)
- [Semi-Supervised Learning for Eye Image Segmentation](https://mlsanity.com/paper/2103.09369/) (2021)
- [HOIGaze: Gaze Estimation During Hand-Object Interactions in Extended Reality Exploiting Eye-Hand-Head Coordination](https://mlsanity.com/paper/2504.19828/) (2025)
- [PupilSense: A Novel Application for Webcam-Based Pupil Diameter Estimation](https://mlsanity.com/paper/2407.11204/) (2024)
- [Mazed and Confused: A Dataset of Cybersickness, Working Memory, Mental Load, Physical Load, and Attention During a Real Walking Task in VR](https://mlsanity.com/paper/2409.06898/) (2024)
- [Towards Unsupervised Eye-Region Segmentation for Eye Tracking](https://mlsanity.com/paper/2410.06131/) (2024)
- [Zero-Shot Pupil Segmentation with SAM 2: A Case Study of Over 14 Million Images](https://mlsanity.com/paper/2410.08926/) (2024)
- [FOVAL: Calibration-Free and Subject-Invariant Fixation Depth Estimation Across Diverse Eye-Tracking Datasets](https://mlsanity.com/paper/2408.03591/) (2024)
- [DeepACTIF: Efficient Feature Attribution via Activation Traces in Neural Sequence Models](https://mlsanity.com/paper/2509.19362/) (2025)
- [Seeing My Future: Predicting Situated Interaction Behavior in Virtual Reality](https://mlsanity.com/paper/2510.10742/) (2025)
- [Realizing Fully-Integrated, Low-Power, Event-Based Pupil Tracking with Neuromorphic Hardware](https://mlsanity.com/paper/2511.20175/) (2025)
- [Eye image segmentation using visual and concept prompts with Segment Anything Model 3 (SAM3)](https://mlsanity.com/paper/2603.17715/) (2026)
- [A Multi-Modal Dataset for Ground Reaction Force Estimation Using Consumer Wearable Sensors](https://mlsanity.com/paper/2603.28784/) (2026)
- [Data-driven Head Motion Generation through Natural Gaze-Head Coordination](https://mlsanity.com/paper/2605.25810/) (2026)

## Abstract (arXiv)

The interaction between the vestibular and ocular system has primarily been studied in controlled environments. Consequently, off-the shelf tools for categorization of gaze events (e.g. fixations, pursuits, saccade) fail when head movements are allowed. Our approach was to collect a novel, naturalistic, and multimodal dataset of eye+head movements when subjects performed everyday tasks while wearing a mobile eye tracker equipped with an inertial measurement unit and a 3D stereo camera. This Gaze-in-the-Wild dataset (GW) includes eye+head rotational velocities (deg/s), infrared eye images and scene imagery (RGB+D). A portion was labelled by coders into gaze motion events with a mutual agreement of 0.72 sample based Cohen's kappa. This labelled data was used to train and evaluate two machine learning algorithms, Random Forest and a Recurrent Neural Network model, for gaze event classification. Assessment involved the application of established and novel event based performance metrics. Classifiers achieve ~90% human performance in detecting fixations and saccades but fall short (60%) on detecting pursuit movements. Moreover, pursuit classification is far worse in the absence of head movement information. A subsequent analysis of feature significance in our best-performing model revealed a reliance upon absolute eye and head velocity, indicating that classification does not require spatial alignment of the head and eye tracking coordinate systems. The GW dataset, trained classifiers and evaluation metrics will be made publicly available with the intention of facilitating growth in the emerging area of head-free gaze event classification.