# Wearable Accelerometer Foundation Models for Health via Knowledge Distillation

arXiv: https://arxiv.org/abs/2412.11276 · HTML: https://mlsanity.com/paper/2412.11276/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 64 references that are in the map.
- First posted on arXiv 15 Dec 2024.

## Builds on (most cited first, 30 of 45)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [Cross Modal Distillation for Supervision Transfer](https://mlsanity.com/paper/1507.00448/) (2015)
- [CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients](https://mlsanity.com/paper/2005.13249/) (2020)
- [SEED: Self-supervised Distillation For Visual Representation](https://mlsanity.com/paper/2101.04731/) (2021)
- [An Empirical Survey of Data Augmentation for Time Series Classification with Neural Networks](https://mlsanity.com/paper/2007.15951/) (2020)
- [4M: Massively Multimodal Masked Modeling](https://mlsanity.com/paper/2312.06647/) (2023)
- [Spreading vectors for similarity search](https://mlsanity.com/paper/1806.03198/) (2018)
- [SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding](https://mlsanity.com/paper/2310.15308/) (2023)
- [Self-supervised ECG Representation Learning for Emotion Recognition](https://mlsanity.com/paper/2002.03898/) (2020)
- [Large-scale Training of Foundation Models for Wearable Biosignals](https://mlsanity.com/paper/2312.05409/) (2023)
- [Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation](https://mlsanity.com/paper/2205.14141/) (2022)
- [Self-supervised representation learning from 12-lead ECG data](https://mlsanity.com/paper/2103.12676/) (2021)
- [Subject-Aware Contrastive Learning for Biosignals](https://mlsanity.com/paper/2007.04871/) (2020)
- [Exploring Contrastive Learning in Human Activity Recognition for Healthcare](https://mlsanity.com/paper/2011.11542/) (2020)

## Built on by (most cited first, 16 of 16)

- [Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictions](https://mlsanity.com/paper/2507.00191/) (2025)
- [Wavelet-Driven Masked Multiscale Reconstruction for PPG Foundation Models](https://mlsanity.com/paper/2601.12215/) (2026)
- [A robust PPG foundation model using multimodal physiological supervision](https://mlsanity.com/paper/2606.07365/) (2026)
- [SlotFM: A Motion Foundation Model with Slot Attention for Diverse Downstream Tasks](https://mlsanity.com/paper/2509.21673/) (2025)
- [Learning Clinical Representations Under Systematic Distribution Shift](https://mlsanity.com/paper/2603.07348/) (2026)
- [Discriminative Representation Learning for Clinical Prediction](https://mlsanity.com/paper/2603.20921/) (2026)
- [Sonata: A Hybrid World Model for Inertial Kinematics under Clinical Data Scarcity](https://mlsanity.com/paper/2604.18058/) (2026)
- [OpenMHC: Accelerating the Science of Wearable Foundation Models](https://mlsanity.com/paper/2607.16235/) (2026)
- [Learning Longitudinal Health Representations from EHR and Wearable Data](https://mlsanity.com/paper/2601.12227/) (2026)
- [Uncertainty-Aware Foundation Models for Clinical Data](https://mlsanity.com/paper/2604.04175/) (2026)
- [PULSE: Privileged Knowledge Transfer from Rich to Deployable Sensors for Embodied Multi-Sensory Learning](https://mlsanity.com/paper/2510.24058/) (2025)
- [Emergent Symbolic Structure in Health Foundation Models: Extraction, Alignment, and Cross-Modal Transfer](https://mlsanity.com/paper/2605.07407/) (2026)
- [WavesFM: Hierarchical Representation Learning for Longitudinal Wearable Sensor Waveforms](https://mlsanity.com/paper/2605.09173/) (2026)
- [BCG-FM: A Foundation Model for Ambient Cardiac Health Sensing](https://mlsanity.com/paper/2606.07692/) (2026)
- [BRIDGE-EEG: Bridging Self-Supervised Pretraining and Efficient Deployment for Cross-Dataset EEG Classification](https://mlsanity.com/paper/2609.12218/) (2026)
- [A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensions](https://mlsanity.com/paper/2609.33727/) (2026)

## Abstract (arXiv)

Modern wearable devices can conveniently record various biosignals in the many different environments of daily living, enabling a rich view of individual health. However, not all biosignals are the same: high-fidelity biosignals, such as photoplethysmogram (PPG), contain more physiological information, but require optical sensors with a high power footprint. Alternatively, a lower-fidelity biosignal such as accelerometry has a significantly smaller power footprint and is available in almost any wearable device. While accelerometry is widely used for activity recognition and fitness, it is less explored for health biomarkers and diagnosis. Here, we show that an accelerometry foundation model can predict a wide variety of health targets. To achieve improved performance, we distill representational knowledge from PPG encoders to accelerometery encoders using 20 million minutes of unlabeled data, collected from ~172K participants in the Apple Heart and Movement Study under informed consent. We observe strong cross-modal alignment on unseen data, e.g., 99.2% top-1 accuracy for retrieving PPG embeddings from accelerometry embeddings. We show that distilled accelerometry encoders have significantly more informative representations compared to self-supervised or supervised encoders trained directly on accelerometry data, observed by at least 23%-49% improved performance for predicting heart rate and heart rate variability. We also show that distilled accelerometry encoders are readily predictive of a wide array of downstream health targets, i.e., they are generalist foundation models. We believe accelerometry foundation models for health may unlock new opportunities for developing digital biomarkers from any wearable device.