# Hallucinated Heartbeats: Anomaly-Aware Remote Pulse Estimation

arXiv: https://arxiv.org/abs/2303.06452 · HTML: https://mlsanity.com/paper/2303.06452/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 47 references that are in the map.
- First posted on arXiv 11 Mar 2023.

## Builds on (most cited first, 17 of 17)

- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [The Deepfake Detection Challenge (DFDC) Preview Dataset](https://mlsanity.com/paper/1910.08854/) (2019)
- [DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks](https://mlsanity.com/paper/1805.07888/) (2018)
- [Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement](https://mlsanity.com/paper/2006.03790/) (2020)
- [The DeepFake Detection Challenge (DFDC) Dataset](https://mlsanity.com/paper/2006.07397/) (2020)
- [Remote Photoplethysmograph Signal Measurement from Facial Videos Using Spatio-Temporal Networks](https://mlsanity.com/paper/1905.02419/) (2019)
- [PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer](https://mlsanity.com/paper/2111.12082/) (2021)
- [RhythmNet: End-to-end Heart Rate Estimation from Face via Spatial-temporal Representation](https://mlsanity.com/paper/1910.11515/) (2019)
- [Remote Heart Rate Measurement from Highly Compressed Facial Videos: an End-to-end Deep Learning Solution with Video Enhancement](https://mlsanity.com/paper/1907.11921/) (2019)
- [VIPL-HR: A Multi-modal Database for Pulse Estimation from Less-constrained Face Video](https://mlsanity.com/paper/1810.04927/) (2018)
- [Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-Learner](https://mlsanity.com/paper/2007.06786/) (2020)
- [Camera Measurement of Physiological Vital Signs](https://mlsanity.com/paper/2111.11547/) (2021)
- [Deception Detection and Remote Physiological Monitoring: A Dataset and Baseline Experimental Results](https://mlsanity.com/paper/2106.06583/) (2021)
- [Remote Photoplethysmography: Rarely Considered Factors](https://mlsanity.com/paper/2004.12695/) (2020)
- [Perfusion assessment via local remote photoplethysmography (rPPG)](https://mlsanity.com/paper/2208.13840/) (2022)
- [Digital and Physical-World Attacks on Remote Pulse Detection](https://mlsanity.com/paper/2110.11525/) (2021)

## Built on by (most cited first, 1 of 1)

- [SiNC+: Adaptive Camera-Based Vitals with Unsupervised Learning of Periodic Signals](https://mlsanity.com/paper/2404.13449/) (2024)

## Abstract (arXiv)

Camera-based physiological monitoring, especially remote photoplethysmography (rPPG), is a promising tool for health diagnostics, and state-of-the-art pulse estimators have shown impressive performance on benchmark datasets. We argue that evaluations of modern solutions may be incomplete, as we uncover failure cases for videos without a live person, or in the presence of severe noise. We demonstrate that spatiotemporal deep learning models trained only with live samples "hallucinate" a genuine-shaped pulse on anomalous and noisy videos, which may have negative consequences when rPPG models are used by medical personnel. To address this, we offer: (a) An anomaly detection model, built on top of the predicted waveforms. We compare models trained in open-set (unknown abnormal predictions) and closed-set (abnormal predictions known when training) settings; (b) An anomaly-aware training regime that penalizes the model for predicting periodic signals from anomalous videos. Extensive experimentation with eight research datasets (rPPG-specific: DDPM, CDDPM, PURE, UBFC, ARPM; deep fakes: DFDC; face presentation attack detection: HKBU-MARs; rPPG outlier: KITTI) show better accuracy of anomaly detection for deep learning models incorporating the proposed training (75.8%), compared to models trained regularly (73.7%) and to hand-crafted rPPG methods (52-62%).