# NeuroLip: An Event-driven Spatiotemporal Learning Framework for Cross-Scene Lip-Motion-based Visual Speaker Recognition

arXiv: https://arxiv.org/abs/2604.15718 · HTML: https://mlsanity.com/paper/2604.15718/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 72 references that are in the map.
- First posted on arXiv 17 Apr 2026.

## Builds on (most cited first, 24 of 24)

- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [End-to-End Learning of Representations for Asynchronous Event-Based Data](https://mlsanity.com/paper/1904.08245/) (2019)
- [TAM: Temporal Adaptive Module for Video Recognition](https://mlsanity.com/paper/2005.06803/) (2020)
- [UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer](https://mlsanity.com/paper/2211.09552/) (2022)
- [Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis](https://mlsanity.com/paper/1704.04086/) (2017)
- [Time-Ordered Recent Event (TORE) Volumes for Event Cameras](https://mlsanity.com/paper/2103.06108/) (2021)
- [GET: Group Event Transformer for Event-Based Vision](https://mlsanity.com/paper/2310.02642/) (2023)
- [Temporal Interlacing Network](https://mlsanity.com/paper/2001.06499/) (2020)
- [SpikePoint: An Efficient Point-based Spiking Neural Network for Event Cameras Action Recognition](https://mlsanity.com/paper/2310.07189/) (2023)
- [Hypergraph-based Multi-View Action Recognition using Event Cameras](https://mlsanity.com/paper/2403.19316/) (2024)
- [End-to-End Multi-View Lipreading](https://mlsanity.com/paper/1709.00443/) (2017)
- [EgoEvGesture: Gesture Recognition Based on Egocentric Event Camera](https://mlsanity.com/paper/2503.12419/) (2025)
- [PAM: Pose Attention Module for Pose-Invariant Face Recognition](https://mlsanity.com/paper/2111.11940/) (2021)
- [MTGA: Multi-View Temporal Granularity Aligned Aggregation for Event-Based Lip-Reading](https://mlsanity.com/paper/2404.11979/) (2024)
- [One-shot lip-based biometric authentication: extending behavioral features with authentication phrase information](https://mlsanity.com/paper/2308.06944/) (2023)
- [A Framework for Event-based Computer Vision on a Mobile Device](https://mlsanity.com/paper/2205.06836/) (2022)
- [WhisperNetV2: SlowFast Siamese Network For Lip-Based Biometrics](https://mlsanity.com/paper/2407.08717/) (2024)
- [DynamicLip: Shape-Independent Continuous Authentication via Lip Articulator Dynamics](https://mlsanity.com/paper/2501.01032/) (2025)

## Abstract (arXiv)

Visual speaker recognition based on lip motion offers a silent, hands-free, and behavior-driven biometric solution that remains effective even when acoustic cues are unavailable. Compared to traditional methods that rely heavily on appearance-dependent representations, lip motion encodes subject-specific behavioral dynamics driven by consistent articulation patterns and muscle coordination, offering inherent stability across environmental changes. However, capturing these robust, fine-grained dynamics is challenging for conventional frame-based cameras due to motion blur and low dynamic range. To exploit the intrinsic stability of lip motion and address these sensing limitations, we propose NeuroLip, an event-based framework that captures fine-grained lip dynamics under a strict yet practical cross-scene protocol: training is performed under a single controlled condition, while recognition must generalize to unseen viewing and lighting conditions. NeuroLip features a 1) Temporal-aware Voxel Encoding module with adaptive event weighting, 2) Structure-aware Spatial Enhancer that amplifies discriminative behavioral patterns by suppressing noise while preserving vertically structured motion information, and 3) Polarity Consistency Regularization mechanism to retain motion-direction cues encoded in event polarities. To facilitate systematic evaluation, we introduce DVSpeaker, a comprehensive event-based lip-motion dataset comprising 50 subjects recorded under four distinct viewpoint and illumination scenarios. Extensive experiments demonstrate that NeuroLip achieves near-perfect matched-scene accuracy and robust cross-scene generalization, attaining over 71% accuracy on unseen viewpoints and nearly 76% under low-light conditions, outperforming representative existing methods by at least 8.54%. The dataset and code are publicly available at https://github.com/JiuZeongit/NeuroLip.