# Remote Heart Rate Monitoring in Smart Environments from Videos with Self-supervised Pre-training

arXiv: https://arxiv.org/abs/2310.15388 · HTML: https://mlsanity.com/paper/2310.15388/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 88 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2104.14548/) (2021)
- [Deep Face Recognition: A Survey](https://mlsanity.com/paper/1804.06655/) (2018)
- [FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding](https://mlsanity.com/paper/2103.05950/) (2021)
- [DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks](https://mlsanity.com/paper/1805.07888/) (2018)
- [Remote Photoplethysmograph Signal Measurement from Facial Videos Using Spatio-Temporal Networks](https://mlsanity.com/paper/1905.02419/) (2019)
- [Self-supervised ECG Representation Learning for Emotion Recognition](https://mlsanity.com/paper/2002.03898/) (2020)
- [Self-Supervised Learning for Videos: A Survey](https://mlsanity.com/paper/2207.00419/) (2022)
- [MedAug: Contrastive learning leveraging patient metadata improves representations for chest X-ray interpretation](https://mlsanity.com/paper/2102.10663/) (2021)
- [A Reproducible Study on Remote Heart Rate Measurement](https://mlsanity.com/paper/1709.00962/) (2017)
- [Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition](https://mlsanity.com/paper/2008.09412/) (2020)
- [Self-supervised Learning for ECG-based Emotion Recognition](https://mlsanity.com/paper/1910.07497/) (2019)
- [Learning Motion-Robust Remote Photoplethysmography through Arbitrary Resolution Videos](https://mlsanity.com/paper/2211.16922/) (2022)
- [Self-supervised Contrastive Learning of Multi-view Facial Expressions](https://mlsanity.com/paper/2108.06723/) (2021)
- [Dual-path TokenLearner for Remote Photoplethysmography-based Physiological Measurement with Facial Videos](https://mlsanity.com/paper/2308.07771/) (2023)
- [Analysis of Augmentations for Contrastive ECG Representation Learning](https://mlsanity.com/paper/2206.07656/) (2022)
- [Spatiotemporal Contrastive Learning of Facial Expressions in Videos](https://mlsanity.com/paper/2108.03064/) (2021)

## Built on by (most cited first, 1 of 1)

- [Beyond LoRA: Exploring Efficient Fine-Tuning Techniques for Time Series Foundational Models](https://mlsanity.com/paper/2409.11302/) (2024)

## Abstract (arXiv)

Recent advances in deep learning have made it increasingly feasible to estimate heart rate remotely in smart environments by analyzing videos. However, a notable limitation of deep learning methods is their heavy reliance on extensive sets of labeled data for effective training. To address this issue, self-supervised learning has emerged as a promising avenue. Building on this, we introduce a solution that utilizes self-supervised contrastive learning for the estimation of remote photoplethysmography (PPG) and heart rate monitoring, thereby reducing the dependence on labeled data and enhancing performance. We propose the use of 3 spatial and 3 temporal augmentations for training an encoder through a contrastive framework, followed by utilizing the late-intermediate embeddings of the encoder for remote PPG and heart rate estimation. Our experiments on two publicly available datasets showcase the improvement of our proposed approach over several related works as well as supervised learning baselines, as our results approach the state-of-the-art. We also perform thorough experiments to showcase the effects of using different design choices such as the video representation learning method, the augmentations used in the pre-training stage, and others. We also demonstrate the robustness of our proposed method over the supervised learning approaches on reduced amounts of labeled data.