# Neural Networks for Semantic Gaze Analysis in XR Settings

arXiv: https://arxiv.org/abs/2103.10451 · HTML: https://mlsanity.com/paper/2103.10451/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 32 references that are in the map.
- First posted on arXiv 18 Mar 2021.

## Builds on (most cited first, 6 of 6)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [Fixing the train-test resolution discrepancy](https://mlsanity.com/paper/1906.06423/) (2019)
- [What Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation?](https://mlsanity.com/paper/1801.06397/) (2018)

## Built on by (most cited first, 2 of 2)

- [Vision-language models for decoding provider attention during neonatal resuscitation](https://mlsanity.com/paper/2404.01207/) (2024)
- [The eyes and hearts of UAV pilots: observations of physiological responses in real-life scenarios](https://mlsanity.com/paper/2210.14910/) (2022)

## Abstract (arXiv)

Virtual-reality (VR) and augmented-reality (AR) technology is increasingly combined with eye-tracking. This combination broadens both fields and opens up new areas of application, in which visual perception and related cognitive processes can be studied in interactive but still well controlled settings. However, performing a semantic gaze analysis of eye-tracking data from interactive three-dimensional scenes is a resource-intense task, which so far has been an obstacle to economic use. In this paper we present a novel approach which minimizes time and information necessary to annotate volumes of interest (VOIs) by using techniques from object recognition. To do so, we train convolutional neural networks (CNNs) on synthetic data sets derived from virtual models using image augmentation techniques. We evaluate our method in real and virtual environments, showing that the method can compete with state-of-the-art approaches, while not relying on additional markers or preexisting databases but instead offering cross-platform use.