# LLaSA: A Sensor-Aware LLM for Natural Language Reasoning of Human Activity from IMU Data

arXiv: https://arxiv.org/abs/2406.14498 · HTML: https://mlsanity.com/paper/2406.14498/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 13 of its 38 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 13 of 13)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents](https://mlsanity.com/paper/2311.05437/) (2023)
- [Platypus: Quick, Cheap, and Powerful Refinement of LLMs](https://mlsanity.com/paper/2308.07317/) (2023)
- [Large Language Models for Time Series: A Survey](https://mlsanity.com/paper/2402.01801/) (2024)
- [LLM-based NLG Evaluation: Current Status and Challenges](https://mlsanity.com/paper/2402.01383/) (2024)
- [Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Data](https://mlsanity.com/paper/2401.06866/) (2024)
- [ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding System](https://mlsanity.com/paper/2304.14407/) (2023)
- [LLMSense: Harnessing LLMs for High-level Reasoning Over Spatiotemporal Sensor Traces](https://mlsanity.com/paper/2403.19857/) (2024)
- [Towards a Personal Health Large Language Model](https://mlsanity.com/paper/2406.06474/) (2024)
- [Transforming Wearable Data into Personal Health Insights using Large Language Model Agents](https://mlsanity.com/paper/2406.06464/) (2024)

## Built on by (most cited first, 11 of 11)

- [ZARA: Training-Free Motion Time-Series Reasoning via Evidence-Grounded LLM Agents](https://mlsanity.com/paper/2508.04038/) (2025)
- [LLaVA-SLT: Visual Language Tuning for Sign Language Translation](https://mlsanity.com/paper/2412.16524/) (2024)
- [OWL: Geometry-Aware Spatial Reasoning for Audio Large Language Models](https://mlsanity.com/paper/2509.26140/) (2025)
- [Exploring the Capabilities of LLMs for IMU-based Fine-grained Human Activity Understanding](https://mlsanity.com/paper/2504.02878/) (2025)
- [Using LLMs for Late Multimodal Sensor Fusion for Activity Recognition](https://mlsanity.com/paper/2509.10729/) (2025)
- [ConSensus: Multi-Agent Collaboration for Multimodal Sensing](https://mlsanity.com/paper/2601.06453/) (2026)
- [AnyMo: Geometry-Aware Setup-Agnostic Modeling of Human Motion in the Wild](https://mlsanity.com/paper/2605.22715/) (2026)
- [HALO: A Heterogeneity-Aware Language-Aligned IMU Foundation Model for Open-Set Human Activity Recognition](https://mlsanity.com/paper/2608.27233/) (2026)
- [SensingAgents: A Multi-Agent Collaborative Framework for Robust IMU Activity Recognition](https://mlsanity.com/paper/2605.04608/) (2026)
- [TimeBraid: Unifying Time Series and Language for Understanding and Forecasting](https://mlsanity.com/paper/2609.29792/) (2026)
- [SenseAgent: An LLM Agent for Adaptive Cross-Domain IMU Sensing](https://mlsanity.com/paper/2609.32000/) (2026)

## Abstract (arXiv)

Wearable systems can recognize activities from IMU data but often fail to explain their underlying causes or contextual significance. To address this limitation, we introduce two large-scale resources: SensorCap, comprising 35,960 IMU--caption pairs, and OpenSQA, with 199,701 question--answer pairs designed for causal and explanatory reasoning. OpenSQA includes a curated tuning split (Tune-OpenSQA) optimized for scientific accuracy, narrative clarity, and diagnostic insight. Leveraging these datasets, we develop LLaSA (Large Language and Sensor Assistant), a family of compact sensor-aware language models (7B and 13B) that generate interpretable, context-rich responses to open-ended questions grounded in raw IMU data. LLaSA outperforms commercial LLMs, including GPT-3.5 and GPT-4o-mini, on benchmark and real-world tasks, demonstrating the effectiveness of domain supervision and model alignment for sensor reasoning. Our code repository and datasets can be found at https://github.com/BASHLab/LLaSA.