# PhysFormer++: Facial Video-based Physiological Measurement with SlowFast Temporal Difference Transformer

arXiv: https://arxiv.org/abs/2302.03548 · HTML: https://mlsanity.com/paper/2302.03548/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 93 references that are in the map.
- First posted on arXiv 7 Feb 2023.

## Builds on (most cited first, 30 of 61)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [TransReID: Transformer-based Object Re-Identification](https://mlsanity.com/paper/2102.04378/) (2021)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)

## Built on by (most cited first, 30 of 48)

- [rPPG-MAE: Self-supervised Pre-training with Masked Autoencoders for Remote Physiological Measurement](https://mlsanity.com/paper/2306.02301/) (2023)
- [Contrast-Phys+: Unsupervised and Weakly-supervised Video-based Remote Physiological Measurement via Spatiotemporal Contrast](https://mlsanity.com/paper/2309.06924/) (2023)
- [RhythmFormer: Extracting Patterned rPPG Signals based on Periodic Sparse Attention](https://mlsanity.com/paper/2402.12788/) (2024)
- [GPT as Psychologist? Preliminary Evaluations for GPT-4V on Visual Affective Computing](https://mlsanity.com/paper/2403.05916/) (2024)
- [RhythmMamba: Fast, Lightweight, and Accurate Remote Physiological Measurement](https://mlsanity.com/paper/2404.06483/) (2024)
- [Spiking-PhysFormer: Camera-Based Remote Photoplethysmography with Parallel Spike-driven Transformer](https://mlsanity.com/paper/2402.04798/) (2024)
- [PhysMamba: Efficient Remote Physiological Measurement with SlowFast Temporal Difference Mamba](https://mlsanity.com/paper/2409.12031/) (2024)
- [FactorizePhys: Matrix Factorization for Multidimensional Attention in Remote Physiological Sensing](https://mlsanity.com/paper/2411.01542/) (2024)
- [BigSmall: Efficient Multi-Task Learning for Disparate Spatial and Temporal Physiological Measurements](https://mlsanity.com/paper/2303.11573/) (2023)
- [Advancing Generalizable Remote Physiological Measurement through the Integration of Explicit and Implicit Prior Knowledge](https://mlsanity.com/paper/2403.06947/) (2024)
- [PhysMLE: Generalizable and Priors-Inclusive Multi-task Remote Physiological Measurement](https://mlsanity.com/paper/2405.06201/) (2024)
- [Efficient Mixture-of-Expert for Video-based Driver State and Physiological Multi-task Estimation in Conditional Autonomous Driving](https://mlsanity.com/paper/2410.21086/) (2024)
- [CardiacMamba: A Multimodal RGB-RF Fusion Framework with State Space Models for Remote Physiological Measurement](https://mlsanity.com/paper/2502.13624/) (2025)
- [PhysDrive: A Multimodal Remote Physiological Measurement Dataset for In-vehicle Driver Monitoring](https://mlsanity.com/paper/2507.19172/) (2025)
- [Remote Photoplethysmography in Real-World and Extreme Lighting Scenarios](https://mlsanity.com/paper/2503.11465/) (2025)
- [FreqPhys: Repurposing Implicit Physiological Frequency Prior for Robust Remote Photoplethysmography](https://mlsanity.com/paper/2604.00534/) (2026)
- [Align the GAP: Prior-based Unified Multi-Task Remote Physiological Measurement Framework For Domain Generalization and Personalization](https://mlsanity.com/paper/2506.16160/) (2025)
- [PhysFlow: Skin tone transfer for remote heart rate estimation through conditional normalizing flows](https://mlsanity.com/paper/2407.21519/) (2024)
- [Semi-rPPG: Semi-Supervised Remote Physiological Measurement with Curriculum Pseudo-Labeling](https://mlsanity.com/paper/2502.03855/) (2025)
- [Not Only Consistency: Enhance Test-Time Adaptation with Spatio-temporal Inconsistency for Remote Physiological Measurement](https://mlsanity.com/paper/2507.07908/) (2025)
- [Reperio-rPPG: Relational Temporal Graph Neural Networks for Periodicity Learning in Remote Physiological Measurement](https://mlsanity.com/paper/2511.05946/) (2025)
- [Phase-shifted remote photoplethysmography for estimating heart rate and blood pressure from facial video](https://mlsanity.com/paper/2401.04560/) (2024)
- [Deep adaptative spectral zoom for improved remote heart rate estimation](https://mlsanity.com/paper/2403.06902/) (2024)
- [PhysMamba: State Space Duality Model for Remote Physiological Measurement](https://mlsanity.com/paper/2408.01077/) (2024)
- [BeatFormer: Efficient motion-robust remote heart rate estimation through unsupervised spectral zoomed attention filters](https://mlsanity.com/paper/2507.14885/) (2025)
- [PhysNeXt: Next-Generation Dual-Branch Structured Attention Fusion Network for Remote Photoplethysmography Measurement](https://mlsanity.com/paper/2603.19752/) (2026)
- [Remote Bio-Sensing: Open Source Benchmark Framework for Fair Evaluation of rPPG](https://mlsanity.com/paper/2307.12644/) (2023)
- [Periodic-MAE: Periodic Video Masked Autoencoder for rPPG Estimation](https://mlsanity.com/paper/2506.21855/) (2025)
- [Editing Physiological Signals in Videos Using Latent Representations](https://mlsanity.com/paper/2509.25348/) (2025)
- [KDPhys: An Attention Guided 3D to 2D Knowledge Distillation for Real-time Video-Based Physiological Measurement](https://mlsanity.com/paper/2601.00714/) (2026)

## Abstract (arXiv)

Remote photoplethysmography (rPPG), which aims at measuring heart activities and physiological signals from facial video without any contact, has great potential in many applications (e.g., remote healthcare and affective computing). Recent deep learning approaches focus on mining subtle rPPG clues using convolutional neural networks with limited spatio-temporal receptive fields, which neglect the long-range spatio-temporal perception and interaction for rPPG modeling. In this paper, we propose two end-to-end video transformer based architectures, namely PhysFormer and PhysFormer++, to adaptively aggregate both local and global spatio-temporal features for rPPG representation enhancement. As key modules in PhysFormer, the temporal difference transformers first enhance the quasi-periodic rPPG features with temporal difference guided global attention, and then refine the local spatio-temporal representation against interference. To better exploit the temporal contextual and periodic rPPG clues, we also extend the PhysFormer to the two-pathway SlowFast based PhysFormer++ with temporal difference periodic and cross-attention transformers. Furthermore, we propose the label distribution learning and a curriculum learning inspired dynamic constraint in frequency domain, which provide elaborate supervisions for PhysFormer and PhysFormer++ and alleviate overfitting. Comprehensive experiments are performed on four benchmark datasets to show our superior performance on both intra- and cross-dataset testings. Unlike most transformer networks needed pretraining from large-scale datasets, the proposed PhysFormer family can be easily trained from scratch on rPPG datasets, which makes it promising as a novel transformer baseline for the rPPG community.