# SensorLM: Learning the Language of Wearable Sensors

arXiv: https://arxiv.org/abs/2506.09108 · HTML: https://mlsanity.com/paper/2506.09108/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 50 references that are in the map.
- First posted on arXiv 10 Jun 2025.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://mlsanity.com/paper/2108.10904/) (2021)
- [Masked Siamese Networks for Label-Efficient Learning](https://mlsanity.com/paper/2204.07141/) (2022)
- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [Large Language Models are Few-Shot Health Learners](https://mlsanity.com/paper/2305.15525/) (2023)
- [Large-scale Training of Foundation Models for Wearable Biosignals](https://mlsanity.com/paper/2312.05409/) (2023)
- [Image Captioners Are Scalable Vision Learners Too](https://mlsanity.com/paper/2306.07915/) (2023)
- [Language Models Still Struggle to Zero-shot Reason about Time Series](https://mlsanity.com/paper/2404.11757/) (2024)
- [NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals](https://mlsanity.com/paper/2409.00101/) (2024)
- [Exploring Contrastive Learning in Human Activity Recognition for Healthcare](https://mlsanity.com/paper/2011.11542/) (2020)
- [Self-supervised Learning for Human Activity Recognition Using 700,000 Person-days of Wearable Data](https://mlsanity.com/paper/2206.02909/) (2022)
- [Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Data](https://mlsanity.com/paper/2401.06866/) (2024)
- [Scaling Wearable Foundation Models](https://mlsanity.com/paper/2410.13638/) (2024)
- [SimPer: Simple Self-Supervised Learning of Periodic Targets](https://mlsanity.com/paper/2210.03115/) (2022)
- [SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals](https://mlsanity.com/paper/2405.17766/) (2024)
- [ECG-Chat: A Large ECG-Language Model for Cardiac Disease Diagnosis](https://mlsanity.com/paper/2408.08849/) (2024)
- [Data augmentation for learning predictive models on EEG: a systematic comparison](https://mlsanity.com/paper/2206.14483/) (2022)
- [Towards a Personal Health Large Language Model](https://mlsanity.com/paper/2406.06474/) (2024)
- [UniMTS: Unified Pre-training for Motion Time Series](https://mlsanity.com/paper/2410.19818/) (2024)
- [SensorChat: Answering Qualitative and Quantitative Questions during Long-Term Multimodal Sensor Interactions](https://mlsanity.com/paper/2502.02883/) (2025)
- [Sensor2Text: Enabling Natural Language Interactions for Daily Activity Tracking Using Wearable Sensors](https://mlsanity.com/paper/2410.20034/) (2024)
- [Guidelines for Augmentation Selection in Contrastive Learning for Time Series Classification](https://mlsanity.com/paper/2407.09336/) (2024)

## Built on by (most cited first, 30 of 49)

- [OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data](https://mlsanity.com/paper/2510.02410/) (2025)
- [ZARA: Training-Free Motion Time-Series Reasoning via Evidence-Grounded LLM Agents](https://mlsanity.com/paper/2508.04038/) (2025)
- [SleepLM: Natural-Language Intelligence for Human Sleep](https://mlsanity.com/paper/2602.23605/) (2026)
- [OSF: On Pre-training and Scaling of Sleep Foundation Models](https://mlsanity.com/paper/2603.00190/) (2026)
- [HEARTS: Benchmarking LLM Reasoning on Health Time Series](https://mlsanity.com/paper/2603.06638/) (2026)
- [GlucoFM: A Dual-Stream Foundation Model for Continuous Glucose Monitoring](https://mlsanity.com/paper/2605.30865/) (2026)
- [Towards Generalizable Human Activity Recognition: A Survey](https://mlsanity.com/paper/2508.12213/) (2025)
- [Using LLMs for Late Multimodal Sensor Fusion for Activity Recognition](https://mlsanity.com/paper/2509.10729/) (2025)
- [Use of Continuous Glucose Monitoring with Machine Learning to Identify Metabolic Subphenotypes and Inform Precision Lifestyle Changes](https://mlsanity.com/paper/2511.03986/) (2025)
- [How Well Do Multimodal Models Reason on ECG Signals?](https://mlsanity.com/paper/2603.00312/) (2026)
- [PulseLM: A Foundation Dataset and Benchmark for PPG-Text Learning](https://mlsanity.com/paper/2603.03331/) (2026)
- [Wearable Foundation Models Should Go Beyond Static Encoders](https://mlsanity.com/paper/2603.19564/) (2026)
- [A robust PPG foundation model using multimodal physiological supervision](https://mlsanity.com/paper/2606.07365/) (2026)
- [Inertia-1: An Open Exploration of Wearable Motion Foundation Models](https://mlsanity.com/paper/2607.06617/) (2026)
- [HERMES: A Unified Open-Source Framework for Realtime Multimodal Physiological Sensing, Edge AI, and Intervention in Closed-Loop Smart Healthcare Applications](https://mlsanity.com/paper/2601.12610/) (2026)
- [Sonata: A Hybrid World Model for Inertial Kinematics under Clinical Data Scarcity](https://mlsanity.com/paper/2604.18058/) (2026)
- [Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUs](https://mlsanity.com/paper/2604.21926/) (2026)
- [Signal or Noise? Understanding Generative Models for Real-World Sensor Time Series](https://mlsanity.com/paper/2607.04245/) (2026)
- [AnyMo: Geometry-Aware Setup-Agnostic Modeling of Human Motion in the Wild](https://mlsanity.com/paper/2605.22715/) (2026)
- [LLM-Powered Personalized Glycemic Assessment in Type 2 Diabetes with Wearable Sensor Data](https://mlsanity.com/paper/2606.12699/) (2026)
- [Agents in the Large: Perception-Centered Architecture for Persistent Agents](https://mlsanity.com/paper/2608.30478/) (2026)
- [FM-FoG: A Real-Time Foundation Model-based Wearable System for Freezing-of-Gait Mitigation](https://mlsanity.com/paper/2509.24176/) (2025)
- [Prototype-based Self-Supervised Multimodal Learning for PPG and Accelerometry Signals](https://mlsanity.com/paper/2510.09764/) (2025)
- [Towards Characterizing Knowledge Distillation of PPG Heart Rate Estimation Models](https://mlsanity.com/paper/2511.18829/) (2025)
- [VitalDiagnosis: AI-Driven Ecosystem for 24/7 Vital Monitoring and Chronic Disease Management](https://mlsanity.com/paper/2601.15798/) (2026)
- [Physiology as Language: Translating Respiration to Sleep EEG](https://mlsanity.com/paper/2602.00526/) (2026)
- [DomusFM: A Foundation Model for Event-Based Behavioral Monitoring in Smart-Homes](https://mlsanity.com/paper/2602.01910/) (2026)
- [Exploring AI-Augmented Sensemaking of Patient-Generated Health Data: A Mixed-Method Study with Healthcare Professionals in Cardiac Risk Reduction](https://mlsanity.com/paper/2602.05687/) (2026)
- [ActivityNarrated: An Open-Ended Narrative Paradigm for Wearable Human Activity Understanding](https://mlsanity.com/paper/2604.00767/) (2026)
- [RF-LEGO: Modularized Signal Processing-Deep Learning Co-Design for RF Sensing via Deep Unrolling](https://mlsanity.com/paper/2604.10183/) (2026)

## Abstract (arXiv)

We present SensorLM, a family of sensor-language foundation models that enable wearable sensor data understanding with natural language. Despite its pervasive nature, aligning and interpreting sensor data with language remains challenging due to the lack of paired, richly annotated sensor-text descriptions in uncurated, real-world wearable data. We introduce a hierarchical caption generation pipeline designed to capture statistical, structural, and semantic information from sensor data. This approach enabled the curation of the largest sensor-language dataset to date, comprising over 59.7 million hours of data from more than 103,000 people. Furthermore, SensorLM extends prominent multimodal pretraining architectures (e.g., CLIP, CoCa) and recovers them as specific variants within a generic architecture. Extensive experiments on real-world tasks in human activity analysis and healthcare verify the superior performance of SensorLM over state-of-the-art in zero-shot recognition, few-shot learning, and cross-modal retrieval. SensorLM also demonstrates intriguing capabilities including scaling behaviors, label efficiency, sensor captioning, and zero-shot generalization to unseen tasks.