# Training Robust Deep Physiological Measurement Models with Synthetic Video-based Data

arXiv: https://arxiv.org/abs/2311.05371 · HTML: https://mlsanity.com/paper/2311.05371/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 60 references that are in the map.
- First posted on arXiv 9 Nov 2023.

## Builds on (most cited first, 30 of 31)

- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [Time Series Data Augmentation for Deep Learning: A Survey](https://mlsanity.com/paper/2002.12478/) (2020)
- [Driving in the Matrix: Can Virtual Worlds Replace Human-Generated Annotations for Real World Tasks?](https://mlsanity.com/paper/1610.01983/) (2016)
- [Playing for Benchmarks](https://mlsanity.com/paper/1709.07322/) (2017)
- [DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks](https://mlsanity.com/paper/1805.07888/) (2018)
- [Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement](https://mlsanity.com/paper/2006.03790/) (2020)
- [Data Augmentation of Wearable Sensor Data for Parkinson's Disease Monitoring using Convolutional Neural Networks](https://mlsanity.com/paper/1706.00527/) (2017)
- [Synthetic Data for Deep Learning](https://mlsanity.com/paper/1909.11512/) (2019)
- [Multi-Scale Convolutional Neural Networks for Time Series Classification](https://mlsanity.com/paper/1603.06995/) (2016)
- [EfficientPhys: Enabling Simple, Fast and Accurate Camera-Based Vitals Measurement](https://mlsanity.com/paper/2110.04447/) (2021)
- [C2N: Practical Generative Noise Modeling for Real-World Denoising](https://mlsanity.com/paper/2202.09533/) (2022)
- [RobustTAD: Robust Time Series Anomaly Detection via Decomposition and Convolutional Neural Networks](https://mlsanity.com/paper/2002.09545/) (2020)
- [VideoMix: Rethinking Data Augmentation for Video Classification](https://mlsanity.com/paper/2012.03457/) (2020)
- [A Reproducible Study on Remote Heart Rate Measurement](https://mlsanity.com/paper/1709.00962/) (2017)
- [Camera Measurement of Physiological Vital Signs](https://mlsanity.com/paper/2111.11547/) (2021)
- [SimPer: Simple Self-Supervised Learning of Periodic Targets](https://mlsanity.com/paper/2210.03115/) (2022)
- [Effective Use of Synthetic Data for Urban Scene Semantic Segmentation](https://mlsanity.com/paper/1807.06132/) (2018)
- [AutoHR: A Strong End-to-end Baseline for Remote Heart Rate Measurement with Neural Searching](https://mlsanity.com/paper/2004.12292/) (2020)
- [rPPG-MAE: Self-supervised Pre-training with Masked Autoencoders for Remote Physiological Measurement](https://mlsanity.com/paper/2306.02301/) (2023)
- [When do GANs replicate? On the choice of dataset size](https://mlsanity.com/paper/2202.11765/) (2022)
- [SCAMPS: Synthetics for Camera Measurement of Physiological Signals](https://mlsanity.com/paper/2206.04197/) (2022)
- [Synthetic Data in Human Analysis: A Survey](https://mlsanity.com/paper/2208.09191/) (2022)
- [Pixel-wise Crowd Understanding via Synthetic Data](https://mlsanity.com/paper/2007.16032/) (2020)
- [On the Applicability of Synthetic Data for Face Recognition](https://mlsanity.com/paper/2104.02815/) (2021)
- [Mixing Real and Synthetic Data to Enhance Neural Network Training -- A Review of Current Approaches](https://mlsanity.com/paper/2007.08781/) (2020)
- [VersatileGait: A Large-Scale Synthetic Gait Dataset with Fine-GrainedAttributes and Complicated Scenarios](https://mlsanity.com/paper/2101.01394/) (2021)
- [SynSig2Vec: Learning Representations from Synthetic Dynamic Signatures for Real-world Verification](https://mlsanity.com/paper/1911.05358/) (2019)
- [Synthetic Data for Multi-Parameter Camera-Based Physiological Sensing](https://mlsanity.com/paper/2110.04902/) (2021)

## Abstract (arXiv)

Recent advances in supervised deep learning techniques have demonstrated the possibility to remotely measure human physiological vital signs (e.g., photoplethysmograph, heart rate) just from facial videos. However, the performance of these methods heavily relies on the availability and diversity of real labeled data. Yet, collecting large-scale real-world data with high-quality labels is typically challenging and resource intensive, which also raises privacy concerns when storing personal bio-metric data. Synthetic video-based datasets (e.g., SCAMPS citemcduff2022scamps) with photo-realistic synthesized avatars are introduced to alleviate the issues while providing high-quality synthetic data. However, there exists a significant gap between synthetic and real-world data, which hinders the generalization of neural models trained on these synthetic datasets. In this paper, we proposed several measures to add real-world noise to synthetic physiological signals and corresponding facial videos. We experimented with individual and combined augmentation methods and evaluated our framework on three public real-world datasets. Our results show that we were able to reduce the average MAE from 6.9 to 2.0.