# Spiking-PhysFormer: Camera-Based Remote Photoplethysmography with Parallel Spike-driven Transformer

arXiv: https://arxiv.org/abs/2402.04798 · HTML: https://mlsanity.com/paper/2402.04798/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 99 references that are in the map.
- First posted on arXiv 7 Feb 2024.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Deep Residual Learning in Spiking Neural Networks](https://mlsanity.com/paper/2102.04159/) (2021)
- [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022)
- [SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence](https://mlsanity.com/paper/2310.16620/) (2023)
- [DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks](https://mlsanity.com/paper/1805.07888/) (2018)
- [Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement](https://mlsanity.com/paper/2006.03790/) (2020)
- [PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer](https://mlsanity.com/paper/2111.12082/) (2021)
- [Temporal-wise Attention Spiking Neural Networks for Event Streams Classification](https://mlsanity.com/paper/2107.11711/) (2021)
- [Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks](https://mlsanity.com/paper/2003.06696/) (2020)
- [Remote Heart Rate Measurement from Highly Compressed Facial Videos: an End-to-end Deep Learning Solution with Video Enhancement](https://mlsanity.com/paper/1907.11921/) (2019)
- [Video-based Remote Physiological Measurement via Cross-verified Feature Disentangling](https://mlsanity.com/paper/2007.08213/) (2020)
- [EfficientPhys: Enabling Simple, Fast and Accurate Camera-Based Vitals Measurement](https://mlsanity.com/paper/2110.04447/) (2021)
- [rPPG-Toolbox: Deep Remote PPG Toolbox](https://mlsanity.com/paper/2210.00716/) (2022)
- [Segment Any Anomaly without Training via Hybrid Prompt Regularization](https://mlsanity.com/paper/2305.10724/) (2023)
- [MMPD: Multi-Domain Mobile Video Physiology Dataset](https://mlsanity.com/paper/2302.03840/) (2023)
- [Camera Measurement of Physiological Vital Signs](https://mlsanity.com/paper/2111.11547/) (2021)
- [PhysFormer++: Facial Video-based Physiological Measurement with SlowFast Temporal Difference Transformer](https://mlsanity.com/paper/2302.03548/) (2023)
- [AutoHR: A Strong End-to-end Baseline for Remote Heart Rate Measurement with Neural Searching](https://mlsanity.com/paper/2004.12292/) (2020)
- [Simplifying Transformer Blocks](https://mlsanity.com/paper/2311.01906/) (2023)
- [Optical flow estimation from event-based cameras and spiking neural networks](https://mlsanity.com/paper/2302.06492/) (2023)
- [Learning Motion-Robust Remote Photoplethysmography through Arbitrary Resolution Videos](https://mlsanity.com/paper/2211.16922/) (2022)
- [Competition-Level Problems are Effective LLM Evaluators](https://mlsanity.com/paper/2312.02143/) (2023)
- [Facial Video-based Remote Physiological Measurement via Self-supervised Learning](https://mlsanity.com/paper/2210.15401/) (2022)
- [SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition](https://mlsanity.com/paper/2308.04369/) (2023)
- [A Transformer-based deep neural network model for SSVEP classification](https://mlsanity.com/paper/2210.04172/) (2022)
- [Highly Efficient 3D Human Pose Tracking from Events with Spiking Spatiotemporal Transformer](https://mlsanity.com/paper/2303.09681/) (2023)
- [Spiking Generative Adversarial Networks With a Neural Network Discriminator: Local Training, Bayesian Models, and Continual Meta-Learning](https://mlsanity.com/paper/2111.01750/) (2021)
- [SparseSpikformer: A Co-Design Framework for Token and Weight Pruning in Spiking Transformer](https://mlsanity.com/paper/2311.08806/) (2023)

## Built on by (most cited first, 19 of 19)

- [RhythmFormer: Extracting Patterned rPPG Signals based on Periodic Sparse Attention](https://mlsanity.com/paper/2402.12788/) (2024)
- [RhythmMamba: Fast, Lightweight, and Accurate Remote Physiological Measurement](https://mlsanity.com/paper/2404.06483/) (2024)
- [PhysMamba: Efficient Remote Physiological Measurement with SlowFast Temporal Difference Mamba](https://mlsanity.com/paper/2409.12031/) (2024)
- [Toward Large-scale Spiking Neural Networks: A Comprehensive Survey and Future Directions](https://mlsanity.com/paper/2409.02111/) (2024)
- [PhysDrive: A Multimodal Remote Physiological Measurement Dataset for In-vehicle Driver Monitoring](https://mlsanity.com/paper/2507.19172/) (2025)
- [Remote Photoplethysmography in Real-World and Extreme Lighting Scenarios](https://mlsanity.com/paper/2503.11465/) (2025)
- [Camera-Based Remote Physiology Sensing for Hundreds of Subjects Across Skin Tones](https://mlsanity.com/paper/2404.05003/) (2024)
- [Non-Contact Health Monitoring During Daily Personal Care Routines](https://mlsanity.com/paper/2506.09718/) (2025)
- [Memory-efficient Low-latency Remote Photoplethysmography through Temporal-Spatial State Space Duality](https://mlsanity.com/paper/2504.01774/) (2025)
- [BeatFormer: Efficient motion-robust remote heart rate estimation through unsupervised spectral zoomed attention filters](https://mlsanity.com/paper/2507.14885/) (2025)
- [Exploring Reliable PPG Authentication on Smartwatches in Daily Scenarios](https://mlsanity.com/paper/2503.23930/) (2025)
- [M3PD Dataset: Dual-view Photoplethysmography (PPG) Using Front-and-rear Cameras of Smartphones in Lab and Clinical Settings](https://mlsanity.com/paper/2511.02349/) (2025)
- [PhysFlow: Frequency Decoupled with Dual-Field Rectified Flow for Remote Photoplethysmography](https://mlsanity.com/paper/2606.23226/) (2026)
- [Self-cross Feature based Spiking Neural Networks for Efficient Few-shot Learning](https://mlsanity.com/paper/2505.07921/) (2025)
- [Exploring Remote Physiological Signal Measurement under Dynamic Lighting Conditions at Night: Dataset, Experiment, and Analysis](https://mlsanity.com/paper/2507.04306/) (2025)
- [FacePhys: State of the Heart Learning](https://mlsanity.com/paper/2512.06275/) (2025)
- [Facial Spatiotemporal Graphs: Leveraging the 3D Facial Surface for Remote Physiological Measurement](https://mlsanity.com/paper/2601.13724/) (2026)
- [LQ-rPPG: A Label-Quantized Coarse-to-Fine Learning Framework for Remote Physiological Measurement](https://mlsanity.com/paper/2605.23174/) (2026)
- [StreamPPG: Low-Latency rPPG Estimation via Consistent Privileged Learning](https://mlsanity.com/paper/2606.23186/) (2026)

## Abstract (arXiv)

Artificial neural networks (ANNs) can help camera-based remote photoplethysmography (rPPG) in measuring cardiac activity and physiological signals from facial videos, such as pulse wave, heart rate and respiration rate with better accuracy. However, most existing ANN-based methods require substantial computing resources, which poses challenges for effective deployment on mobile devices. Spiking neural networks (SNNs), on the other hand, hold immense potential for energy-efficient deep learning owing to their binary and event-driven architecture. To the best of our knowledge, we are the first to introduce SNNs into the realm of rPPG, proposing a hybrid neural network (HNN) model, the Spiking-PhysFormer, aimed at reducing power consumption. Specifically, the proposed Spiking-PhyFormer consists of an ANN-based patch embedding block, SNN-based transformer blocks, and an ANN-based predictor head. First, to simplify the transformer block while preserving its capacity to aggregate local and global spatio-temporal features, we design a parallel spike transformer block to replace sequential sub-blocks. Additionally, we propose a simplified spiking self-attention mechanism that omits the value parameter without compromising the model's performance. Experiments conducted on four datasets-PURE, UBFC-rPPG, UBFC-Phys, and MMPD demonstrate that the proposed model achieves a 12.4% reduction in power consumption compared to PhysFormer. Additionally, the power consumption of the transformer block is reduced by a factor of 12.2, while maintaining decent performance as PhysFormer and other ANN-based models.