# Merging Multiple Datasets for Improved Appearance-Based Gaze Estimation

arXiv: https://arxiv.org/abs/2409.00912 · HTML: https://mlsanity.com/paper/2409.00912/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 2 Sep 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [HOTR: End-to-End Human-Object Interaction Detection with Transformers](https://mlsanity.com/paper/2104.13682/) (2021)
- [Eye Tracking for Everyone](https://mlsanity.com/paper/1606.05814/) (2016)
- [Appearance-Based Gaze Estimation in the Wild](https://mlsanity.com/paper/1504.02863/) (2015)
- [ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variation](https://mlsanity.com/paper/2007.15837/) (2020)
- [It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation](https://mlsanity.com/paper/1611.08860/) (2016)
- [MSeg: A Composite Dataset for Multi-domain Semantic Segmentation](https://mlsanity.com/paper/2112.13762/) (2021)
- [Detecting Attended Visual Targets in Video](https://mlsanity.com/paper/2003.02501/) (2020)
- [Unified Quality Assessment of In-the-Wild Videos with Mixed Datasets Training](https://mlsanity.com/paper/2011.04263/) (2020)
- [Gaze Estimation using Transformer](https://mlsanity.com/paper/2105.14424/) (2021)
- [Deep Pictorial Gaze Estimation](https://mlsanity.com/paper/1807.10002/) (2018)
- [Transformer-based Multimodal Information Fusion for Facial Expression Analysis](https://mlsanity.com/paper/2203.12367/) (2022)
- [End-to-End Human-Gaze-Target Detection with Transformers](https://mlsanity.com/paper/2203.10433/) (2022)
- [Appearance-Based Gaze Estimation Using Dilated-Convolutions](https://mlsanity.com/paper/1903.07296/) (2019)
- [Cross-Dataset Person Re-Identification via Unsupervised Pose Disentanglement and Adaptation](https://mlsanity.com/paper/1909.09675/) (2019)
- [A Differential Approach for Gaze Estimation](https://mlsanity.com/paper/1904.09459/) (2019)
- [Grapy-ML: Graph Pyramid Mutual Learning for Cross-dataset Human Parsing](https://mlsanity.com/paper/1911.12053/) (2019)
- [Unsupervised Cross-dataset Person Re-identification by Transfer Learning of Spatial-Temporal Patterns](https://mlsanity.com/paper/1803.07293/) (2018)
- [Gaze Estimation with an Ensemble of Four Architectures](https://mlsanity.com/paper/2107.01980/) (2021)
- [Using Variable Dwell Time to Accelerate Gaze-Based Web Browsing with Two-Step Selection](https://mlsanity.com/paper/1704.06399/) (2017)

## Built on by (most cited first, 2 of 2)

- [GazeVLM: A Vision-Language Model for Multi-Task Gaze Understanding](https://mlsanity.com/paper/2511.06348/) (2025)
- [DMAGaze: Gaze Estimation Based on Feature Disentanglement and Multi-Scale Attention](https://mlsanity.com/paper/2504.11160/) (2025)

## Abstract (arXiv)

Multiple datasets have been created for training and testing appearance-based gaze estimators. Intuitively, more data should lead to better performance. However, combining datasets to train a single esti-mator rarely improves gaze estimation performance. One reason may be differences in the experimental protocols used to obtain the gaze sam-ples, resulting in differences in the distributions of head poses, gaze an-gles, illumination, etc. Another reason may be the inconsistency between methods used to define gaze angles (label mismatch). We propose two innovations to improve the performance of gaze estimation by leveraging multiple datasets, a change in the estimator architecture and the intro-duction of a gaze adaptation module. Most state-of-the-art estimators merge information extracted from images of the two eyes and the entire face either in parallel or combine information from the eyes first then with the face. Our proposed Two-stage Transformer-based Gaze-feature Fusion (TTGF) method uses transformers to merge information from each eye and the face separately and then merge across the two eyes. We argue that this improves head pose invariance since changes in head pose affect left and right eye images in different ways. Our proposed Gaze Adaptation Module (GAM) method handles annotation inconsis-tency by applying a Gaze Adaption Module for each dataset to correct gaze estimates from a single shared estimator. This enables us to combine information across datasets despite differences in labeling. Our experi-ments show that these innovations improve gaze estimation performance over the SOTA both individually and collectively (by 10% - 20%). Our code is available at https://github.com/HKUST-NISL/GazeSetMerge.