# Sensor-Language-Action Models

arXiv: https://arxiv.org/abs/2610.08244 · HTML: https://mlsanity.com/paper/2610.08244/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 43 references that are in the map.
- First posted on arXiv 6 Oct 2026.

## Builds on (most cited first, 16 of 16)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [Chronos-2: From Univariate to Universal Forecasting](https://mlsanity.com/paper/2510.15821/) (2025)
- [SensorLM: Learning the Language of Wearable Sensors](https://mlsanity.com/paper/2506.09108/) (2025)
- [OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data](https://mlsanity.com/paper/2510.02410/) (2025)
- [SleepLM: Natural-Language Intelligence for Human Sleep](https://mlsanity.com/paper/2602.23605/) (2026)
- [OSF: On Pre-training and Scaling of Sleep Foundation Models](https://mlsanity.com/paper/2603.00190/) (2026)
- [HEARTS: Benchmarking LLM Reasoning on Health Time Series](https://mlsanity.com/paper/2603.06638/) (2026)
- [GlucoFM: A Dual-Stream Foundation Model for Continuous Glucose Monitoring](https://mlsanity.com/paper/2605.30865/) (2026)
- [MetaboNet: The Largest Publicly Available Consolidated Dataset for Type 1 Diabetes Management](https://mlsanity.com/paper/2601.11505/) (2026)
- [Inertia-1: An Open Exploration of Wearable Motion Foundation Models](https://mlsanity.com/paper/2607.06617/) (2026)
- [Signal or Noise? Understanding Generative Models for Real-World Sensor Time Series](https://mlsanity.com/paper/2607.04245/) (2026)

## Abstract (arXiv)

Sensors are useful not only for understanding the world but also for deciding what to do next. Existing sensor models however largely stop at perception: they recognize states or predict outcomes, leaving actions modeled separately through task-specific and often closed label spaces. We introduce Sensor-Language-Action (SLA) modeling, a framework that connects multimodal sensor observations, natural language, and actions within a unified model. SLA uses language as a semantic interface between sensing and acting, allowing heterogeneous actions to be represented, predicted, and explained while remaining grounded in the underlying sensor evidence. We build a large-scale SLA benchmark consisting of datasets that span more than 116,000 individuals, 79 sensor modalities, and 60 action groups, together with a multi-faceted captioning pipeline that aligns user context, sensor dynamics, and action evidence. Building on this framework, we present OpenSLA, a unified SLA model for hierarchical action prediction, state understanding, and action explanation. Extensive experiments on real-world tasks in clinical prediction, operating rooms, and metabolic health verify its superior performance over the state-of-the-art. OpenSLA also demonstrates intriguing capabilities including language-guided evidence grounding and zero-shot generalization to unseen actions and cohorts.