# RhythmFormer: Extracting Patterned rPPG Signals based on Periodic Sparse Attention

arXiv: https://arxiv.org/abs/2402.12788 · HTML: https://mlsanity.com/paper/2402.12788/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 55 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [MaxViT: Multi-Axis Vision Transformer](https://mlsanity.com/paper/2204.01697/) (2022)
- [TDN: Temporal Difference Networks for Efficient Action Recognition](https://mlsanity.com/paper/2012.10071/) (2020)
- [DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks](https://mlsanity.com/paper/1805.07888/) (2018)
- [BiFormer: Vision Transformer with Bi-Level Routing Attention](https://mlsanity.com/paper/2303.08810/) (2023)
- [Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement](https://mlsanity.com/paper/2006.03790/) (2020)
- [Remote Photoplethysmograph Signal Measurement from Facial Videos Using Spatio-Temporal Networks](https://mlsanity.com/paper/1905.02419/) (2019)
- [PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer](https://mlsanity.com/paper/2111.12082/) (2021)
- [RhythmNet: End-to-end Heart Rate Estimation from Face via Spatial-temporal Representation](https://mlsanity.com/paper/1910.11515/) (2019)
- [Shunted Self-Attention via Multi-Scale Token Aggregation](https://mlsanity.com/paper/2111.15193/) (2021)
- [Video-based Remote Physiological Measurement via Cross-verified Feature Disentangling](https://mlsanity.com/paper/2007.08213/) (2020)
- [EfficientPhys: Enabling Simple, Fast and Accurate Camera-Based Vitals Measurement](https://mlsanity.com/paper/2110.04447/) (2021)
- [rPPG-Toolbox: Deep Remote PPG Toolbox](https://mlsanity.com/paper/2210.00716/) (2022)
- [PulseGAN: Learning to generate realistic pulse waveforms in remote photoplethysmography](https://mlsanity.com/paper/2006.02699/) (2020)
- [Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-Learner](https://mlsanity.com/paper/2007.06786/) (2020)
- [MMPD: Multi-Domain Mobile Video Physiology Dataset](https://mlsanity.com/paper/2302.03840/) (2023)
- [A Reproducible Study on Remote Heart Rate Measurement](https://mlsanity.com/paper/1709.00962/) (2017)
- [Camera Measurement of Physiological Vital Signs](https://mlsanity.com/paper/2111.11547/) (2021)
- [PhysFormer++: Facial Video-based Physiological Measurement with SlowFast Temporal Difference Transformer](https://mlsanity.com/paper/2302.03548/) (2023)
- [TransRAC: Encoding Multi-scale Temporal Correlation with Transformers for Repetitive Action Counting](https://mlsanity.com/paper/2204.01018/) (2022)
- [MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection](https://mlsanity.com/paper/2112.03902/) (2021)
- [AutoHR: A Strong End-to-end Baseline for Remote Heart Rate Measurement with Neural Searching](https://mlsanity.com/paper/2004.12292/) (2020)
- [Non-Contrastive Unsupervised Learning of Physiological Signals from Video](https://mlsanity.com/paper/2303.07944/) (2023)
- [rPPG-MAE: Self-supervised Pre-training with Masked Autoencoders for Remote Physiological Measurement](https://mlsanity.com/paper/2306.02301/) (2023)
- [Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition](https://mlsanity.com/paper/2008.09412/) (2020)
- [Contrast-Phys+: Unsupervised and Weakly-supervised Video-based Remote Physiological Measurement via Spatiotemporal Contrast](https://mlsanity.com/paper/2309.06924/) (2023)
- [Learning Motion-Robust Remote Photoplethysmography through Arbitrary Resolution Videos](https://mlsanity.com/paper/2211.16922/) (2022)
- [Facial Video-based Remote Physiological Measurement via Self-supervised Learning](https://mlsanity.com/paper/2210.15401/) (2022)

## Built on by (most cited first, 30 of 30)

- [PHASE-Net: Physics-Grounded Harmonic Attention System for Efficient Remote Photoplethysmography Measurement](https://mlsanity.com/paper/2509.24850/) (2025)
- [Fully Test-Time rPPG Estimation via Synthetic Signal-Guided Feature Learning](https://mlsanity.com/paper/2407.13322/) (2024)
- [FreqPhys: Repurposing Implicit Physiological Frequency Prior for Robust Remote Photoplethysmography](https://mlsanity.com/paper/2604.00534/) (2026)
- [Gaze into the Heart: A Multi-View Video Dataset for rPPG and Health Biomarkers Estimation](https://mlsanity.com/paper/2508.17924/) (2025)
- [Reperio-rPPG: Relational Temporal Graph Neural Networks for Periodicity Learning in Remote Physiological Measurement](https://mlsanity.com/paper/2511.05946/) (2025)
- [Robust and Generalizable Heart Rate Estimation via Deep Learning for Remote Photoplethysmography in Complex Scenarios](https://mlsanity.com/paper/2507.07795/) (2025)
- [PhysNeXt: Next-Generation Dual-Branch Structured Attention Fusion Network for Remote Photoplethysmography Measurement](https://mlsanity.com/paper/2603.19752/) (2026)
- [Periodic-MAE: Periodic Video Masked Autoencoder for rPPG Estimation](https://mlsanity.com/paper/2506.21855/) (2025)
- [M3PD Dataset: Dual-view Photoplethysmography (PPG) Using Front-and-rear Cameras of Smartphones in Lab and Clinical Settings](https://mlsanity.com/paper/2511.02349/) (2025)
- [ToTMNet: FFT-Accelerated Toeplitz Temporal Mixing Network for Lightweight Remote Photoplethysmography](https://mlsanity.com/paper/2601.04159/) (2026)
- [EMPD: An Event-based Multimodal Physiological Dataset for Remote Pulse Wave Detection](https://mlsanity.com/paper/2603.26699/) (2026)
- [HOT: Harmonic-Constrained Optimal Transport for Remote Photoplethysmography Domain Adaptation](https://mlsanity.com/paper/2604.01675/) (2026)
- [PhysFlow: Frequency Decoupled with Dual-Field Rectified Flow for Remote Photoplethysmography](https://mlsanity.com/paper/2606.23226/) (2026)
- [Rhythm-Structured Predictive Learning for Remote Photoplethysmography](https://mlsanity.com/paper/2606.31736/) (2026)
- [Ultra-Short rPPG Estimation via Periodicity Guidance and Signal Reconstruction](https://mlsanity.com/paper/2506.22078/) (2025)
- [Exploring Remote Physiological Signal Measurement under Dynamic Lighting Conditions at Night: Dataset, Experiment, and Analysis](https://mlsanity.com/paper/2507.04306/) (2025)
- [BTS-rPPG: Orthogonal Butterfly Temporal Shifting for Remote Photoplethysmography](https://mlsanity.com/paper/2604.01679/) (2026)
- [Video-based Heart Rate Estimation with Angle-guided ROI Optimization and Graph Signal Denoising](https://mlsanity.com/paper/2604.11395/) (2026)
- [Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuning](https://mlsanity.com/paper/2604.22838/) (2026)
- [LQ-rPPG: A Label-Quantized Coarse-to-Fine Learning Framework for Remote Physiological Measurement](https://mlsanity.com/paper/2605.23174/) (2026)
- [Explaining RhythmFormer: A Systematic XAI Analysis of Periodic Sparse Attention for Remote Photoplethysmography](https://mlsanity.com/paper/2606.13839/) (2026)
- [Fusion-E2Pulse: A Multimodal Event-RGB Fusion Network for Non-contact Pulse Wave Reconstruction](https://mlsanity.com/paper/2606.15597/) (2026)
- [StreamPPG: Low-Latency rPPG Estimation via Consistent Privileged Learning](https://mlsanity.com/paper/2606.23186/) (2026)
- [Physiological Signals as a Forensic Modality for Talking-Face Deepfake Detection](https://mlsanity.com/paper/2607.21776/) (2026)
- [PhysVR: Vision-Language Model Guided Interference-aware Temporal Feature Refinement for Remote Physiological Measurement](https://mlsanity.com/paper/2608.29663/) (2026)
- [Cross-Dataset Transfer and Reliability of Explainable Artificial Intelligence for RhythmFormer Remote Photoplethysmography](https://mlsanity.com/paper/2609.03663/) (2026)
- [Beyond Contact Sensors: Deep learning with Pseudo-Labeling for remote Photoplethysmography](https://mlsanity.com/paper/2609.10026/) (2026)
- [Complementary rPPG-Derived and Lip-Region Frequency Cues for Talking-Face Deepfake Detection](https://mlsanity.com/paper/2609.22284/) (2026)
- [FLOW: Feature-Level Optimal Warping for Generalized Remote Physiological Measurement](https://mlsanity.com/paper/2609.38913/) (2026)
- [Why Do VLMs Fail at Remote Physiological Measurement? Rethinking with Physiological Grounding and Numerical Alignment](https://mlsanity.com/paper/2610.19052/) (2026)

## Abstract (arXiv)

Remote photoplethysmography (rPPG) is a non-contact method for detecting physiological signals based on facial videos, holding high potential in various applications. Due to the periodicity nature of rPPG signals, the long-range dependency capturing capacity of the transformer was assumed to be advantageous for such signals. However, existing methods have not conclusively demonstrated the superior performance of transformers over traditional convolutional neural networks. This may be attributed to the quadratic scaling exhibited by transformer with sequence length, resulting in coarse-grained feature extraction, which in turn affects robustness and generalization. To address that, this paper proposes a periodic sparse attention mechanism based on temporal attention sparsity induced by periodicity. A pre-attention stage is introduced before the conventional attention mechanism. This stage learns periodic patterns to filter out a large number of irrelevant attention computations, thus enabling fine-grained feature extraction. Moreover, to address the issue of fine-grained features being more susceptible to noise interference, a fusion stem is proposed to effectively guide self-attention towards rPPG features. It can be easily integrated into existing methods to enhance their performance. Extensive experiments show that the proposed method achieves state-of-the-art performance in both intra-dataset and cross-dataset evaluations. The codes are available at https://github.com/zizheng-guo/RhythmFormer.