# TransPPG: Two-stream Transformer for Remote Heart Rate Estimate

arXiv: https://arxiv.org/abs/2201.10873 · HTML: https://mlsanity.com/paper/2201.10873/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 38 references that are in the map.
- First posted on arXiv 26 Jan 2022.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks](https://mlsanity.com/paper/1805.07888/) (2018)
- [PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer](https://mlsanity.com/paper/2111.12082/) (2021)
- [RhythmNet: End-to-end Heart Rate Estimation from Face via Spatial-temporal Representation](https://mlsanity.com/paper/1910.11515/) (2019)
- [Remote Heart Rate Measurement from Highly Compressed Facial Videos: an End-to-end Deep Learning Solution with Video Enhancement](https://mlsanity.com/paper/1907.11921/) (2019)
- [Video-based Remote Physiological Measurement via Cross-verified Feature Disentangling](https://mlsanity.com/paper/2007.08213/) (2020)
- [VIPL-HR: A Multi-modal Database for Pulse Estimation from Less-constrained Face Video](https://mlsanity.com/paper/1810.04927/) (2018)
- [AutoHR: A Strong End-to-end Baseline for Remote Heart Rate Measurement with Neural Searching](https://mlsanity.com/paper/2004.12292/) (2020)
- [The Benefit of Distraction: Denoising Remote Vitals Measurements using Inverse Attention](https://mlsanity.com/paper/2010.07770/) (2020)
- [Deep-HR: Fast Heart Rate Estimation from Face Video Under Realistic Conditions](https://mlsanity.com/paper/2002.04821/) (2020)

## Built on by (most cited first, 8 of 8)

- [Self-similarity Prior Distillation for Unsupervised Remote Physiological Measurement](https://mlsanity.com/paper/2311.05100/) (2023)
- [Your smartphone could act as a pulse-oximeter and as a single-lead ECG](https://mlsanity.com/paper/2305.12583/) (2023)
- [PhySU-Net: Long Temporal Context Transformer for rPPG with Self-Supervised Pre-training](https://mlsanity.com/paper/2402.11913/) (2024)
- [DD-rPPGNet: De-interfering and Descriptive Feature Learning for Unsupervised rPPG Estimation](https://mlsanity.com/paper/2407.21402/) (2024)
- [Reperio-rPPG: Relational Temporal Graph Neural Networks for Periodicity Learning in Remote Physiological Measurement](https://mlsanity.com/paper/2511.05946/) (2025)
- [Efficient Deep Learning-based Estimation of the Vital Signs on Smartphones](https://mlsanity.com/paper/2204.08989/) (2022)
- [Generalization of Video-Based Heart Rate Estimation Methods To Low Illumination and Elevated Heart Rates](https://mlsanity.com/paper/2503.11697/) (2025)
- [Non-Contact Physiological Monitoring in Pediatric Intensive Care Units via Adaptive Masking and Self-Supervised Learning](https://mlsanity.com/paper/2602.15967/) (2026)

## Abstract (arXiv)

Non-contact facial video-based heart rate estimation using remote photoplethysmography (rPPG) has shown great potential in many applications (e.g., remote health care) and achieved creditable results in constrained scenarios. However, practical applications require results to be accurate even under complex environment with head movement and unstable illumination. Therefore, improving the performance of rPPG in complex environment has become a key challenge. In this paper, we propose a novel video embedding method that embeds each facial video sequence into a feature map referred to as Multi-scale Adaptive Spatial and Temporal Map with Overlap (MAST_Mop), which contains not only vital information but also surrounding information as reference, which acts as the mirror to figure out the homogeneous perturbations imposed on foreground and background simultaneously, such as illumination instability. Correspondingly, we propose a two-stream Transformer model to map the MAST_Mop into heart rate (HR), where one stream follows the pulse signal in the facial area while the other figures out the perturbation signal from the surrounding region such that the difference of the two channels leads to adaptive noise cancellation. Our approach significantly outperforms all current state-of-the-art methods on two public datasets MAHNOB-HCI and VIPL-HR. As far as we know, it is the first work with Transformer as backbone to capture the temporal dependencies in rPPGs and apply the two stream scheme to figure out the interference from backgrounds as mirror of the corresponding perturbation on foreground signals for noise tolerating.