# rPPG-MAE: Self-supervised Pre-training with Masked Autoencoders for Remote Physiological Measurement

arXiv: https://arxiv.org/abs/2306.02301 · HTML: https://mlsanity.com/paper/2306.02301/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 69 references that are in the map.
- First posted on arXiv 4 Jun 2023.

## Builds on (most cited first, 30 of 38)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers](https://mlsanity.com/paper/2111.12710/) (2021)
- [DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks](https://mlsanity.com/paper/1805.07888/) (2018)
- [Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement](https://mlsanity.com/paper/2006.03790/) (2020)
- [Remote Photoplethysmograph Signal Measurement from Facial Videos Using Spatio-Temporal Networks](https://mlsanity.com/paper/1905.02419/) (2019)
- [PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer](https://mlsanity.com/paper/2111.12082/) (2021)
- [RhythmNet: End-to-end Heart Rate Estimation from Face via Spatial-temporal Representation](https://mlsanity.com/paper/1910.11515/) (2019)
- [Remote Heart Rate Measurement from Highly Compressed Facial Videos: an End-to-end Deep Learning Solution with Video Enhancement](https://mlsanity.com/paper/1907.11921/) (2019)
- [Video-based Remote Physiological Measurement via Cross-verified Feature Disentangling](https://mlsanity.com/paper/2007.08213/) (2020)
- [The Way to my Heart is through Contrastive Learning: Remote Photoplethysmography from Unlabelled Video](https://mlsanity.com/paper/2111.09748/) (2021)
- [PulseGAN: Learning to generate realistic pulse waveforms in remote photoplethysmography](https://mlsanity.com/paper/2006.02699/) (2020)
- [VIPL-HR: A Multi-modal Database for Pulse Estimation from Less-constrained Face Video](https://mlsanity.com/paper/1810.04927/) (2018)
- [Contrast-Phys: Unsupervised Video-based Remote Physiological Measurement via Spatiotemporal Contrast](https://mlsanity.com/paper/2208.04378/) (2022)
- [PhysFormer++: Facial Video-based Physiological Measurement with SlowFast Temporal Difference Transformer](https://mlsanity.com/paper/2302.03548/) (2023)
- [SimPer: Simple Self-Supervised Learning of Periodic Targets](https://mlsanity.com/paper/2210.03115/) (2022)

## Built on by (most cited first, 30 of 37)

- [RhythmFormer: Extracting Patterned rPPG Signals based on Periodic Sparse Attention](https://mlsanity.com/paper/2402.12788/) (2024)
- [A Simple yet Effective Network based on Vision Transformer for Camouflaged Object and Salient Object Detection](https://mlsanity.com/paper/2402.18922/) (2024)
- [GPT as Psychologist? Preliminary Evaluations for GPT-4V on Visual Affective Computing](https://mlsanity.com/paper/2403.05916/) (2024)
- [Advancing Generalizable Remote Physiological Measurement through the Integration of Explicit and Implicit Prior Knowledge](https://mlsanity.com/paper/2403.06947/) (2024)
- [PhysMLE: Generalizable and Priors-Inclusive Multi-task Remote Physiological Measurement](https://mlsanity.com/paper/2405.06201/) (2024)
- [CoEmoGen: Towards Semantically-Coherent and Scalable Emotional Image Content Generation](https://mlsanity.com/paper/2508.03535/) (2025)
- [Orientation-conditioned Facial Texture Mapping for Video-based Facial Remote Photoplethysmography Estimation](https://mlsanity.com/paper/2404.09378/) (2024)
- [Fully Test-Time rPPG Estimation via Synthetic Signal-Guided Feature Learning](https://mlsanity.com/paper/2407.13322/) (2024)
- [Bi-TTA: Bidirectional Test-Time Adapter for Remote Physiological Measurement](https://mlsanity.com/paper/2409.17316/) (2024)
- [Remote Photoplethysmography in Real-World and Extreme Lighting Scenarios](https://mlsanity.com/paper/2503.11465/) (2025)
- [FreqPhys: Repurposing Implicit Physiological Frequency Prior for Robust Remote Photoplethysmography](https://mlsanity.com/paper/2604.00534/) (2026)
- [Semi-rPPG: Semi-Supervised Remote Physiological Measurement with Curriculum Pseudo-Labeling](https://mlsanity.com/paper/2502.03855/) (2025)
- [Not Only Consistency: Enhance Test-Time Adaptation with Spatio-temporal Inconsistency for Remote Physiological Measurement](https://mlsanity.com/paper/2507.07908/) (2025)
- [SiNC+: Adaptive Camera-Based Vitals with Unsupervised Learning of Periodic Signals](https://mlsanity.com/paper/2404.13449/) (2024)
- [Recovering Pulse Waves from Video Using Deep Unrolling and Deep Equilibrium Models](https://mlsanity.com/paper/2503.17269/) (2025)
- [Sage Deer: A Super-Aligned Driving Generalist Is Your Copilot](https://mlsanity.com/paper/2505.10257/) (2025)
- [BeatFormer: Efficient motion-robust remote heart rate estimation through unsupervised spectral zoomed attention filters](https://mlsanity.com/paper/2507.14885/) (2025)
- [PhysNeXt: Next-Generation Dual-Branch Structured Attention Fusion Network for Remote Photoplethysmography Measurement](https://mlsanity.com/paper/2603.19752/) (2026)
- [Periodic-MAE: Periodic Video Masked Autoencoder for rPPG Estimation](https://mlsanity.com/paper/2506.21855/) (2025)
- [HOT: Harmonic-Constrained Optimal Transport for Remote Photoplethysmography Domain Adaptation](https://mlsanity.com/paper/2604.01675/) (2026)
- [Intervention-Based Self-Supervised Learning: A Causal Probe Paradigm for Remote Photoplethysmography](https://mlsanity.com/paper/2605.00882/) (2026)
- [Rhythm-Structured Predictive Learning for Remote Photoplethysmography](https://mlsanity.com/paper/2606.31736/) (2026)
- [Training Robust Deep Physiological Measurement Models with Synthetic Video-based Data](https://mlsanity.com/paper/2311.05371/) (2023)
- [Bootstrapping Vision-language Models for Self-supervised Remote Physiological Measurement](https://mlsanity.com/paper/2407.08507/) (2024)
- [Adversarial Robustness in RGB-Skeleton Action Recognition: Leveraging Attention Modality Reweighter](https://mlsanity.com/paper/2407.19981/) (2024)
- [Time-Series U-Net with Recurrence for Noise-Robust Imaging Photoplethysmography](https://mlsanity.com/paper/2503.17351/) (2025)
- [Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model](https://mlsanity.com/paper/2505.24476/) (2025)
- [Ultra-Short rPPG Estimation via Periodicity Guidance and Signal Reconstruction](https://mlsanity.com/paper/2506.22078/) (2025)
- [CAST-Phys: Contactless Affective States Through Physiological signals Database](https://mlsanity.com/paper/2507.06080/) (2025)
- [Facial Spatiotemporal Graphs: Leveraging the 3D Facial Surface for Remote Physiological Measurement](https://mlsanity.com/paper/2601.13724/) (2026)

## Abstract (arXiv)

Remote photoplethysmography (rPPG) is an important technique for perceiving human vital signs, which has received extensive attention. For a long time, researchers have focused on supervised methods that rely on large amounts of labeled data. These methods are limited by the requirement for large amounts of data and the difficulty of acquiring ground truth physiological signals. To address these issues, several self-supervised methods based on contrastive learning have been proposed. However, they focus on the contrastive learning between samples, which neglect the inherent self-similar prior in physiological signals and seem to have a limited ability to cope with noisy. In this paper, a linear self-supervised reconstruction task was designed for extracting the inherent self-similar prior in physiological signals. Besides, a specific noise-insensitive strategy was explored for reducing the interference of motion and illumination. The proposed framework in this paper, namely rPPG-MAE, demonstrates excellent performance even on the challenging VIPL-HR dataset. We also evaluate the proposed method on two public datasets, namely PURE and UBFC-rPPG. The results show that our method not only outperforms existing self-supervised methods but also exceeds the state-of-the-art (SOTA) supervised methods. One important observation is that the quality of the dataset seems more important than the size in self-supervised pre-training of rPPG. The source code is released at https://github.com/linuxsino/rPPG-MAE.