# Emo-DVS: A Multimodal Benchmark for Privacy-Aware Emotion Recognition with Event Cameras

arXiv: https://arxiv.org/abs/2609.06928 · HTML: https://mlsanity.com/paper/2609.06928/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 55 references that are in the map.
- First posted on arXiv 7 Sep 2026.

## Builds on (most cited first, 18 of 18)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [A Unifying Contrast Maximization Framework for Event Cameras, with Applications to Motion, Depth, and Optical Flow Estimation](https://mlsanity.com/paper/1804.01306/) (2018)
- [Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances](https://mlsanity.com/paper/1905.02947/) (2019)
- [Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs](https://mlsanity.com/paper/1907.06724/) (2019)
- [M3ER: Multiplicative Multimodal Emotion Recognition Using Facial, Textual, and Speech Cues](https://mlsanity.com/paper/1911.05659/) (2019)
- [MGMAE: Motion Guided Masking for Video Masked Autoencoding](https://mlsanity.com/paper/2308.10794/) (2023)
- [Single Image Optical Flow Estimation with an Event Camera](https://mlsanity.com/paper/2004.00347/) (2020)
- [An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos](https://mlsanity.com/paper/2003.00832/) (2020)
- [YOLO-FaceV2: A Scale and Occlusion Aware Face Detector](https://mlsanity.com/paper/2208.02019/) (2022)
- [Heterogeneous Knowledge Transfer in Video Emotion Recognition, Attribution and Summarization](https://mlsanity.com/paper/1511.04798/) (2015)
- [LVFace: Progressive Cluster Optimization for Large Vision Models in Face Recognition](https://mlsanity.com/paper/2501.13420/) (2025)
- [Controllable Augmentations for Video Representation Learning](https://mlsanity.com/paper/2203.16632/) (2022)
- [EZSR: Event-based Zero-Shot Recognition](https://mlsanity.com/paper/2407.21616/) (2024)
- [Neuromorphic Facial Analysis with Cross-Modal Supervision](https://mlsanity.com/paper/2409.10213/) (2024)
- [Robust Facial Reactions Generation: An Emotion-Aware Framework with Modality Compensation](https://mlsanity.com/paper/2407.15798/) (2024)
- [Exploring Spatial-Temporal Dynamics in Event-based Facial Micro-Expression Analysis](https://mlsanity.com/paper/2508.11988/) (2025)
- [CS3D: An Efficient Facial Expression Recognition via Event Vision](https://mlsanity.com/paper/2512.09592/) (2025)

## Abstract (arXiv)

Emotion analysis is a fundamental task in computer vision, but its practical deployment remains constrained by the privacy risks inherent to conventional RGB cameras. Bio-inspired event cameras present a promising hardware-level solution because they capture asynchronous brightness changes, thereby reducing exposure of facial identity details while leveraging high dynamic range for robust perception under challenging illumination conditions. Despite these advantages, existing event-based methods struggle in complex real-world settings due to limited dataset scales, simple acquisition conditions, and reliance on single-modality visual cues. To address these, we establish a challenging tri-modal benchmark with event, audio, and text modalities and propose the Information-Guided Gated Fusion (IGF) framework, which first pre-trains an event encoder on the FAU subset of Emo-DVS to capture fine-grained facial dynamics, then employs adaptive modality gating to suppress modality-specific noise, and finally leverages mutual information maximization to align robust cross-modal representations. To alleviate data scarcity, we introduce Emo-DVS, the first large-scale event-based emotion analysis dataset, which couples dynamic illumination with the Facial Action Unit (FAU) subset and emotion subset. Extensive experiments demonstrate that IGF achieves state-of-the-art performance.