# Joint Spatial-Temporal Modeling and Contrastive Learning for Self-supervised Heart Rate Measurement

arXiv: https://arxiv.org/abs/2406.04942 · HTML: https://mlsanity.com/paper/2406.04942/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 7 Jun 2024.

## Builds on (most cited first, 15 of 15)

- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [RhythmNet: End-to-end Heart Rate Estimation from Face via Spatial-temporal Representation](https://mlsanity.com/paper/1910.11515/) (2019)
- [Remote Heart Rate Measurement from Highly Compressed Facial Videos: an End-to-end Deep Learning Solution with Video Enhancement](https://mlsanity.com/paper/1907.11921/) (2019)
- [Video-based Remote Physiological Measurement via Cross-verified Feature Disentangling](https://mlsanity.com/paper/2007.08213/) (2020)
- [EfficientPhys: Enabling Simple, Fast and Accurate Camera-Based Vitals Measurement](https://mlsanity.com/paper/2110.04447/) (2021)
- [The Way to my Heart is through Contrastive Learning: Remote Photoplethysmography from Unlabelled Video](https://mlsanity.com/paper/2111.09748/) (2021)
- [VIPL-HR: A Multi-modal Database for Pulse Estimation from Less-constrained Face Video](https://mlsanity.com/paper/1810.04927/) (2018)
- [Contrastive Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2211.09980/) (2022)
- [Non-Contrastive Unsupervised Learning of Physiological Signals from Video](https://mlsanity.com/paper/2303.07944/) (2023)
- [Contrast-Phys+: Unsupervised and Weakly-supervised Video-based Remote Physiological Measurement via Spatiotemporal Contrast](https://mlsanity.com/paper/2309.06924/) (2023)
- [EulerMormer: Robust Eulerian Motion Magnification via Dynamic Filtering within Transformer](https://mlsanity.com/paper/2312.04152/) (2023)
- [ViGT: Proposal-free Video Grounding with Learnable Token in Transformer](https://mlsanity.com/paper/2308.06009/) (2023)
- [Dual-path TokenLearner for Remote Photoplethysmography-based Physiological Measurement with Facial Videos](https://mlsanity.com/paper/2308.07771/) (2023)
- [The 1st Challenge on Remote Physiological Signal Sensing (RePSS)](https://mlsanity.com/paper/2003.11756/) (2020)

## Built on by (most cited first, 1 of 1)

- [3D Smoke Scene Reconstruction Guided by Vision Priors from Multimodal Large Language Models](https://mlsanity.com/paper/2604.05687/) (2026)

## Abstract (arXiv)

This paper briefly introduces the solutions developed by our team, HFUT-VUT, for Track 1 of self-supervised heart rate measurement in the 3rd Vision-based Remote Physiological Signal Sensing (RePSS) Challenge hosted at IJCAI 2024. The goal is to develop a self-supervised learning algorithm for heart rate (HR) estimation using unlabeled facial videos. To tackle this task, we present two self-supervised HR estimation solutions that integrate spatial-temporal modeling and contrastive learning, respectively. Specifically, we first propose a non-end-to-end self-supervised HR measurement framework based on spatial-temporal modeling, which can effectively capture subtle rPPG clues and leverage the inherent bandwidth and periodicity characteristics of rPPG to constrain the model. Meanwhile, we employ an excellent end-to-end solution based on contrastive learning, aiming to generalize across different scenarios from complementary perspectives. Finally, we combine the strengths of the above solutions through an ensemble strategy to generate the final predictions, leading to a more accurate HR estimation. As a result, our solutions achieved a remarkable RMSE score of 8.85277 on the test dataset, securing 2nd place in Track 1 of the challenge.