# Evaluating Image-Based Face and Eye Tracking with Event Cameras

arXiv: https://arxiv.org/abs/2408.10395 · HTML: https://mlsanity.com/paper/2408.10395/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 51 references that are in the map.
- First posted on arXiv 19 Aug 2024.

## Builds on (most cited first, 23 of 23)

- [WIDER FACE: A Face Detection Benchmark](https://mlsanity.com/paper/1511.06523/) (2015)
- [Event-based Vision meets Deep Learning on Steering Prediction for Self-driving Cars](https://mlsanity.com/paper/1804.01310/) (2018)
- [Events-to-Video: Bringing Modern Computer Vision to Event Cameras](https://mlsanity.com/paper/1904.08298/) (2019)
- [Time-Ordered Recent Event (TORE) Volumes for Event Cameras](https://mlsanity.com/paper/2103.06108/) (2021)
- [Event Based, Near Eye Gaze Tracking Beyond 10,000Hz](https://mlsanity.com/paper/2004.03577/) (2020)
- [Temporal Binary Representation for Event-Based Action Recognition](https://mlsanity.com/paper/2010.08946/) (2020)
- [EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream](https://mlsanity.com/paper/2012.06475/) (2020)
- [EventSR: From Asynchronous Events to Image Reconstruction, Restoration, and Super-Resolution via End-to-End Adversarial Learning](https://mlsanity.com/paper/2003.07640/) (2020)
- [Towards End-to-end Video-based Eye-Tracking](https://mlsanity.com/paper/2007.13120/) (2020)
- [A Voxel Graph CNN for Object Classification with Event Cameras](https://mlsanity.com/paper/2106.00216/) (2021)
- [3ET: Efficient Event-based Eye Tracking using a Change-Based ConvLSTM Network](https://mlsanity.com/paper/2308.11771/) (2023)
- [Real-Time Face & Eye Tracking and Blink Detection using Event Cameras](https://mlsanity.com/paper/2010.08278/) (2020)
- [Learning from Event Cameras with Sparse Spiking Convolutional Neural Networks](https://mlsanity.com/paper/2104.12579/) (2021)
- [Real-Time Gaze Tracking with Event-Driven Eye Segmentation](https://mlsanity.com/paper/2201.07367/) (2022)
- [Event-Based Eye Tracking. AIS 2024 Challenge Survey](https://mlsanity.com/paper/2404.11770/) (2024)
- [Neuromorphic Event-based Facial Expression Recognition](https://mlsanity.com/paper/2304.06351/) (2023)
- [Event-based Face Detection and Tracking in the Blink of an Eye](https://mlsanity.com/paper/1803.10106/) (2018)
- [Spiking-Fer: Spiking Neural Network for Facial Expression Recognition With Event Cameras](https://mlsanity.com/paper/2304.10211/) (2023)
- [Student Engagement Detection Using Emotion Analysis, Eye Tracking and Head Movement with Machine Learning](https://mlsanity.com/paper/1909.12913/) (2019)
- [Neuromorphic Face Analysis: a Survey](https://mlsanity.com/paper/2402.11631/) (2024)
- [Neuromorphic Valence and Arousal Estimation](https://mlsanity.com/paper/2401.16058/) (2024)
- [Neuromorphic Imaging and Classification with Graph Learning](https://mlsanity.com/paper/2309.15627/) (2023)
- [A Vision Based System for Monitoring the Loss of Attention in Automotive Drivers](https://mlsanity.com/paper/1505.03352/) (2015)

## Built on by (most cited first, 3 of 3)

- [Spatio-temporal Transformers for Action Unit Classification with Event Cameras](https://mlsanity.com/paper/2410.21958/) (2024)
- [Event-based Solutions for Human-centered Applications: A Comprehensive Review](https://mlsanity.com/paper/2502.18490/) (2025)
- [EventFace: Event-Based Face Recognition via Structure-Driven Spatiotemporal Modeling](https://mlsanity.com/paper/2604.06782/) (2026)

## Abstract (arXiv)

Event Cameras, also known as Neuromorphic sensors, capture changes in local light intensity at the pixel level, producing asynchronously generated data termed ``events''. This distinct data format mitigates common issues observed in conventional cameras, like under-sampling when capturing fast-moving objects, thereby preserving critical information that might otherwise be lost. However, leveraging this data often necessitates the development of specialized, handcrafted event representations that can integrate seamlessly with conventional Convolutional Neural Networks (CNNs), considering the unique attributes of event data. In this study, We evaluate event-based Face and Eye tracking. The core objective of our study is to showcase the viability of integrating conventional algorithms with event-based data, transformed into a frame format while preserving the unique benefits of event cameras. To validate our approach, we constructed a frame-based event dataset by simulating events between RGB frames derived from the publicly accessible Helen Dataset. We assess its utility for face and eye detection tasks through the application of GR-YOLO -- a pioneering technique derived from YOLOv3. This evaluation includes a comparative analysis with results derived from training the dataset with YOLOv8. Subsequently, the trained models were tested on real event streams from various iterations of Prophesee's event cameras and further evaluated on the Faces in Event Stream (FES) benchmark dataset. The models trained on our dataset shows a good prediction performance across all the datasets obtained for validation with the best results of a mean Average precision score of 0.91. Additionally, The models trained demonstrated robust performance on real event camera data under varying light conditions.