# In the Blink of an Eye: Event-based Emotion Recognition

arXiv: https://arxiv.org/abs/2310.04043 · HTML: https://mlsanity.com/paper/2310.04043/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 58 references that are in the map.
- First posted on arXiv 6 Oct 2023.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks](https://mlsanity.com/paper/1706.02609/) (2017)
- [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018)
- [End-to-End Learning of Representations for Asynchronous Event-Based Data](https://mlsanity.com/paper/1904.08245/) (2019)
- [Event-based Vision meets Deep Learning on Steering Prediction for Self-driving Cars](https://mlsanity.com/paper/1804.01310/) (2018)
- [EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model](https://mlsanity.com/paper/2205.15278/) (2022)
- [Context-Aware Emotion Recognition Networks](https://mlsanity.com/paper/1908.05913/) (2019)
- [Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction](https://mlsanity.com/paper/2102.09320/) (2021)
- [TransFER: Learning Relation-aware Facial Expression Representations with Transformers](https://mlsanity.com/paper/2108.11116/) (2021)
- [Object Tracking by Jointly Exploiting Frame and Event Domain](https://mlsanity.com/paper/2109.09052/) (2021)
- [Feature Decomposition and Reconstruction Learning for Effective Facial Expression Recognition](https://mlsanity.com/paper/2104.05160/) (2021)
- [Multitask Emotion Recognition with Incomplete Labels](https://mlsanity.com/paper/2002.03557/) (2020)
- [Biologically Inspired Dynamic Thresholds for Spiking Neural Networks](https://mlsanity.com/paper/2206.04426/) (2022)
- [MIMAMO Net: Integrating Micro- and Macro-motion for Video Emotion Recognition](https://mlsanity.com/paper/1911.09784/) (2019)
- [Affective Processes: stochastic modelling of temporal context for emotion and facial expression recognition](https://mlsanity.com/paper/2103.13372/) (2021)
- [Eyemotion: Classifying facial expressions in VR using eye-tracking cameras](https://mlsanity.com/paper/1707.07204/) (2017)
- [Facial Expression Recognition Under Partial Occlusion from Virtual Reality Headsets based on Transfer Learning](https://mlsanity.com/paper/2008.05563/) (2020)
- [Recognizing Facial Expressions of Occluded Faces using Convolutional Neural Networks](https://mlsanity.com/paper/1911.04852/) (2019)

## Built on by (most cited first, 4 of 4)

- [Apprenticeship-Inspired Elegance: Synergistic Knowledge Distillation Empowers Spiking Neural Networks for Efficient Single-Eye Emotion Recognition](https://mlsanity.com/paper/2407.09521/) (2024)
- [EyeTrAES: Fine-grained, Low-Latency Eye Tracking via Adaptive Event Slicing](https://mlsanity.com/paper/2409.18813/) (2024)
- [AnonyNoise: Anonymizing Event Data with Smart Noise to Outsmart Re-Identification and Preserve Privacy](https://mlsanity.com/paper/2411.16440/) (2024)
- [Multi-modal Collaborative Optimization and Expansion Network for Event-assisted Single-eye Expression Recognition](https://mlsanity.com/paper/2505.12007/) (2025)

## Abstract (arXiv)

We introduce a wearable single-eye emotion recognition device and a real-time approach to recognizing emotions from partial observations of an emotion that is robust to changes in lighting conditions. At the heart of our method is a bio-inspired event-based camera setup and a newly designed lightweight Spiking Eye Emotion Network (SEEN). Compared to conventional cameras, event-based cameras offer a higher dynamic range (up to 140 dB vs. 80 dB) and a higher temporal resolution. Thus, the captured events can encode rich temporal cues under challenging lighting conditions. However, these events lack texture information, posing problems in decoding temporal information effectively. SEEN tackles this issue from two different perspectives. First, we adopt convolutional spiking layers to take advantage of the spiking neural network's ability to decode pertinent temporal information. Second, SEEN learns to extract essential spatial cues from corresponding intensity frames and leverages a novel weight-copy scheme to convey spatial attention to the convolutional spiking layers during training and inference. We extensively validate and demonstrate the effectiveness of our approach on a specially collected Single-eye Event-based Emotion (SEE) dataset. To the best of our knowledge, our method is the first eye-based emotion recognition method that leverages event-based cameras and spiking neural network.