mlsanityData to 7 Oct 2026

Wearable Accelerometer Foundation Models for Health via Knowledge Distillation

arXiv 2412.11276 ↗First posted 15 Dec 2024cs.LG cs.AI eess.SPcited by 15

Not on the map. Why?

Abstract (arXiv)

Modern wearable devices can conveniently record various biosignals in the many different environments of daily living, enabling a rich view of individual health. However, not all biosignals are the same: high-fidelity biosignals, such as photoplethysmogram (PPG), contain more physiological information, but require optical sensors with a high power footprint. Alternatively, a lower-fidelity biosignal such as accelerometry has a significantly smaller power footprint and is available in almost any wearable device. While accelerometry is widely used for activity recognition and fitness, it is less explored for health biomarkers and diagnosis. Here, we show that an accelerometry foundation model can predict a wide variety of health targets. To achieve improved performance, we distill representational knowledge from PPG encoders to accelerometery encoders using 20 million minutes of unlabeled data, collected from ~172K participants in the Apple Heart and Movement Study under informed consent. We observe strong cross-modal alignment on unseen data, e.g., 99.2% top-1 accuracy for retrieving PPG embeddings from accelerometry embeddings. We show that distilled accelerometry encoders have significantly more informative representations compared to self-supervised or supervised encoders trained directly on accelerometry data, observed by at least 23%-49% improved performance for predicting heart rate and heart rate variability. We also show that distilled accelerometry encoders are readily predictive of a wide array of downstream health targets, i.e., they are generalist foundation models. We believe accelerometry foundation models for health may unlock new opportunities for developing digital biomarkers from any wearable device.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 45 of its 64 references that are in the map. First posted on arXiv 15 Dec 2024.

Builds on

45 of 64 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  11. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  12. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
33 more
  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  2. 2021Scaling Vision Transformerscited by 993
  3. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  4. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  5. 2021BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG datacited by 194
  6. 2015Cross Modal Distillation for Supervision Transfercited by 173
  7. 2020CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patientscited by 170
  8. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
  9. 2020An Empirical Survey of Data Augmentation for Time Series Classification with Neural Networkscited by 128
  10. 20234M: Massively Multimodal Masked Modelingcited by 118
  11. 2018Spreading vectors for similarity searchcited by 114
  12. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  13. 2020Self-supervised ECG Representation Learning for Emotion Recognitioncited by 99
  14. 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillationcited by 93
  15. 2023Large-scale Training of Foundation Models for Wearable Biosignalscited by 89
  16. 2021Self-supervised representation learning from 12-lead ECG datacited by 80
  17. 2020Subject-Aware Contrastive Learning for Biosignalscited by 75
  18. 2020Exploring Contrastive Learning in Human Activity Recognition for Healthcarecited by 73
  19. 2024Scaling Wearable Foundation Modelscited by 57
  20. 2022What Makes Good Contrastive Learning on Small-Scale Wearable-based Tasks?cited by 39
  21. 2024SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signalscited by 38
  22. 20213KG: Contrastive Learning of 12-Lead Electrocardiograms using Physiologically-Inspired Augmentationscited by 36
  23. 2022IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Textcited by 36
  24. 2020CardioGAN: Attentive Generative Adversarial Network with Dual Discriminators for Synthesis of ECG from PPGcited by 34
  25. 2023EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understandingcited by 32
  26. 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networkscited by 29
  27. 2020Self-supervised transfer learning of physiological representations from free-living wearable datacited by 27
  28. 2023Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignalscited by 22
  29. 2023CroSSL: Cross-modal Self-Supervised Learning for Time-series through Latent Maskingcited by 18
  30. 2023Finding Order in Chaos: A Novel Data Augmentation Method for Time Series in Contrastive Learningcited by 18
  31. 2024Poly-View Contrastive Learningcited by 9
  32. 2020Representation Learning on Variable Length and Incomplete Wearable-Sensory Time Seriescited by 6
  33. 2018Learning Individualized Cardiovascular Responses from Large-scale Wearable Sensors Datacited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 12.

15 papers.

Sort:

By area:

  1. 2025Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictionscited by 18
  2. 2026Wavelet-Driven Masked Multiscale Reconstruction for PPG Foundation Modelscited by 4
  3. 2026A robust PPG foundation model using multimodal physiological supervisioncited by 3
  4. 2026Learning Clinical Representations Under Systematic Distribution Shiftcited by 2
  5. 2026Discriminative Representation Learning for Clinical Predictioncited by 2
  6. 2026Sonata: A Hybrid World Model for Inertial Kinematics under Clinical Data Scarcitycited by 2
  7. 2025SlotFM: A Motion Foundation Model with Slot Attention for Diverse Downstream Taskscited by 1
  8. 2026Learning Longitudinal Health Representations from EHR and Wearable Datacited by 1
  9. 2026Uncertainty-Aware Foundation Models for Clinical Datacited by 1
  10. 2025PULSE: Privileged Knowledge Transfer from Rich to Deployable Sensors for Embodied Multi-Sensory Learningcited by 0
  11. 2026Emergent Symbolic Structure in Health Foundation Models: Extraction, Alignment, and Cross-Modal Transfercited by 0
  12. 2026WavesFM: Hierarchical Representation Learning for Longitudinal Wearable Sensor Waveformscited by 0
3 more
  1. 2026BCG-FM: A Foundation Model for Ambient Cardiac Health Sensingcited by 0
  2. 2026BRIDGE-EEG: Bridging Self-Supervised Pretraining and Efficient Deployment for Cross-Dataset EEG Classificationcited by 0
  3. 2026A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensionscited by 0