# Real-time Multi-person Eyeblink Detection in the Wild for Untrimmed Video

arXiv: https://arxiv.org/abs/2303.16053 · HTML: https://mlsanity.com/paper/2303.16053/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 51 references that are in the map.
- First posted on arXiv 28 Mar 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020)
- [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [Fake It Till You Make It: Face analysis in the wild using synthetic data alone](https://mlsanity.com/paper/2109.15102/) (2021)
- [End-to-end Temporal Action Detection with Transformer](https://mlsanity.com/paper/2106.10271/) (2021)
- [Sample and Computation Redistribution for Efficient Face Detection](https://mlsanity.com/paper/2105.04714/) (2021)
- [MultiSports: A Multi-Person Video Dataset of Spatio-Temporally Localized Sports Actions](https://mlsanity.com/paper/2105.07404/) (2021)
- [Action Tubelet Detector for Spatio-Temporal Action Localization](https://mlsanity.com/paper/1705.01861/) (2017)
- [Actions as Moving Points](https://mlsanity.com/paper/2001.04608/) (2020)
- [TubeR: Tubelet Transformer for Video Action Detection](https://mlsanity.com/paper/2104.00969/) (2021)
- [Temporally Efficient Vision Transformer for Video Instance Segmentation](https://mlsanity.com/paper/2204.08412/) (2022)
- [Efficient Video Instance Segmentation via Tracklet Query and Proposal](https://mlsanity.com/paper/2203.01853/) (2022)
- [A Realistic Dataset and Baseline Temporal Model for Early Drowsiness Detection](https://mlsanity.com/paper/1904.07312/) (2019)
- [mEBAL: A Multimodal Database for Eye Blink Detection and Attention Level Estimation](https://mlsanity.com/paper/2006.05327/) (2020)
- [ALEBk: Feasibility Study of Attention Level Estimation via Blink Detection applied to e-Learning](https://mlsanity.com/paper/2112.09165/) (2021)
- [Towards Real-time Eyeblink Detection in The Wild:Dataset,Theory and Practices](https://mlsanity.com/paper/1902.07891/) (2019)

## Built on by (most cited first, 3 of 3)

- [End-to-end Video Gaze Estimation via Capturing Head-face-eye Spatial-temporal Interaction Context](https://mlsanity.com/paper/2310.18131/) (2023)
- [Reading Between the Frames: Multi-Modal Depression Detection in Videos from Non-Verbal Cues](https://mlsanity.com/paper/2401.02746/) (2024)
- [mEBAL2 Database and Benchmark: Image-based Multispectral Eyeblink Detection](https://mlsanity.com/paper/2309.07880/) (2023)

## Abstract (arXiv)

Real-time eyeblink detection in the wild can widely serve for fatigue detection, face anti-spoofing, emotion analysis, etc. The existing research efforts generally focus on single-person cases towards trimmed video. However, multi-person scenario within untrimmed videos is also important for practical applications, which has not been well concerned yet. To address this, we shed light on this research field for the first time with essential contributions on dataset, theory, and practices. In particular, a large-scale dataset termed MPEblink that involves 686 untrimmed videos with 8748 eyeblink events is proposed under multi-person conditions. The samples are captured from unconstrained films to reveal "in the wild" characteristics. Meanwhile, a real-time multi-person eyeblink detection method is also proposed. Being different from the existing counterparts, our proposition runs in a one-stage spatio-temporal way with end-to-end learning capacity. Specifically, it simultaneously addresses the sub-tasks of face detection, face tracking, and human instance-level eyeblink detection. This paradigm holds 2 main advantages: (1) eyeblink features can be facilitated via the face's global context (e.g., head pose and illumination condition) with joint optimization and interaction, and (2) addressing these sub-tasks in parallel instead of sequential manner can save time remarkably to meet the real-time running requirement. Experiments on MPEblink verify the essential challenges of real-time multi-person eyeblink detection in the wild for untrimmed video. Our method also outperforms existing approaches by large margins and with a high inference speed.