# BigSmall: Efficient Multi-Task Learning for Disparate Spatial and Temporal Physiological Measurements

arXiv: https://arxiv.org/abs/2303.11573 · HTML: https://mlsanity.com/paper/2303.11573/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 67 references that are in the map.
- First posted on arXiv 21 Mar 2023.

## Builds on (most cited first, 26 of 26)

- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [DetCo: Unsupervised Contrastive Learning for Object Detection](https://mlsanity.com/paper/2102.04803/) (2021)
- [DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks](https://mlsanity.com/paper/1805.07888/) (2018)
- [Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement](https://mlsanity.com/paper/2006.03790/) (2020)
- [Remote Photoplethysmograph Signal Measurement from Facial Videos Using Spatio-Temporal Networks](https://mlsanity.com/paper/1905.02419/) (2019)
- [PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer](https://mlsanity.com/paper/2111.12082/) (2021)
- [Hierarchical Multiscale Recurrent Neural Networks](https://mlsanity.com/paper/1609.01704/) (2016)
- [EfficientPhys: Enabling Simple, Fast and Accurate Camera-Based Vitals Measurement](https://mlsanity.com/paper/2110.04447/) (2021)
- [The Way to my Heart is through Contrastive Learning: Remote Photoplethysmography from Unlabelled Video](https://mlsanity.com/paper/2111.09748/) (2021)
- [rPPG-Toolbox: Deep Remote PPG Toolbox](https://mlsanity.com/paper/2210.00716/) (2022)
- [JhatAA-Net: Joint Facial Action Unit Detection and Face Alignment via Adaptive Attention](https://mlsanity.com/paper/2003.08834/) (2020)
- [Deep Adaptive Attention for Joint Facial Action Unit Detection and Face Alignment](https://mlsanity.com/paper/1803.05588/) (2018)
- [Camera Measurement of Physiological Vital Signs](https://mlsanity.com/paper/2111.11547/) (2021)
- [PhysFormer++: Facial Video-based Physiological Measurement with SlowFast Temporal Difference Transformer](https://mlsanity.com/paper/2302.03548/) (2023)
- [Behavioral Use Licensing for Responsible AI](https://mlsanity.com/paper/2011.03116/) (2020)
- [SimPer: Simple Self-Supervised Learning of Periodic Targets](https://mlsanity.com/paper/2210.03115/) (2022)
- [TransRPPG: Remote Photoplethysmography Transformer for 3D Mask Face Presentation Attack Detection](https://mlsanity.com/paper/2104.07419/) (2021)
- [Deep Structure Inference Network for Facial Action Unit Recognition](https://mlsanity.com/paper/1803.05873/) (2018)
- [On Multi-Domain Long-Tailed Recognition, Imbalanced Domain Generalization and Beyond](https://mlsanity.com/paper/2203.09513/) (2022)
- [Active Mini-Batch Sampling using Repulsive Point Processes](https://mlsanity.com/paper/1804.02772/) (2018)
- [Representation Learning with Video Deep InfoMax](https://mlsanity.com/paper/2007.13278/) (2020)
- [The Benefit of Distraction: Denoising Remote Vitals Measurements using Inverse Attention](https://mlsanity.com/paper/2010.07770/) (2020)
- [DeepMag: Source Specific Motion Magnification Using Gradient Ascent](https://mlsanity.com/paper/1808.03338/) (2018)

## Built on by (most cited first, 13 of 13)

- [Scaling Wearable Foundation Models](https://mlsanity.com/paper/2410.13638/) (2024)
- [LSM-2: Learning from Incomplete Wearable Sensor Data](https://mlsanity.com/paper/2506.05321/) (2025)
- [FactorizePhys: Matrix Factorization for Multidimensional Attention in Remote Physiological Sensing](https://mlsanity.com/paper/2411.01542/) (2024)
- [PhysMLE: Generalizable and Priors-Inclusive Multi-task Remote Physiological Measurement](https://mlsanity.com/paper/2405.06201/) (2024)
- [Efficient Mixture-of-Expert for Video-based Driver State and Physiological Multi-task Estimation in Conditional Autonomous Driving](https://mlsanity.com/paper/2410.21086/) (2024)
- [PhysDrive: A Multimodal Remote Physiological Measurement Dataset for In-vehicle Driver Monitoring](https://mlsanity.com/paper/2507.19172/) (2025)
- [Align the GAP: Prior-based Unified Multi-Task Remote Physiological Measurement Framework For Domain Generalization and Personalization](https://mlsanity.com/paper/2506.16160/) (2025)
- [Remote Bio-Sensing: Open Source Benchmark Framework for Fair Evaluation of rPPG](https://mlsanity.com/paper/2307.12644/) (2023)
- [SkinMap: Weighted Full-Body Skin Segmentation for Robust Remote Photoplethysmography](https://mlsanity.com/paper/2510.05296/) (2025)
- [Efficient and Robust Multidimensional Attention in Remote Physiological Sensing through Target Signal Constrained Factorization](https://mlsanity.com/paper/2505.07013/) (2025)
- [Exploring Remote Physiological Signal Measurement under Dynamic Lighting Conditions at Night: Dataset, Experiment, and Analysis](https://mlsanity.com/paper/2507.04306/) (2025)
- [BTS-rPPG: Orthogonal Butterfly Temporal Shifting for Remote Photoplethysmography](https://mlsanity.com/paper/2604.01679/) (2026)
- [A Cloud-Edge System for Multimodal Clinical Screening in Resource-Constrained Rural Settings](https://mlsanity.com/paper/2608.12745/) (2026)

## Abstract (arXiv)

Understanding of human visual perception has historically inspired the design of computer vision architectures. As an example, perception occurs at different scales both spatially and temporally, suggesting that the extraction of salient visual information may be made more effective by paying attention to specific features at varying scales. Visual changes in the body due to physiological processes also occur at different scales and with modality-specific characteristic properties. Inspired by this, we present BigSmall, an efficient architecture for physiological and behavioral measurement. We present the first joint camera-based facial action, cardiac, and pulmonary measurement model. We propose a multi-branch network with wrapping temporal shift modules that yields both accuracy and efficiency gains. We observe that fusing low-level features leads to suboptimal performance, but that fusing high level features enables efficiency gains with negligible loss in accuracy. Experimental results demonstrate that BigSmall significantly reduces the computational costs. Furthermore, compared to existing task-specific models, BigSmall achieves comparable or better results on multiple physiological measurement tasks simultaneously with a unified model.