# Exploring the Capabilities of LLMs for IMU-based Fine-grained Human Activity Understanding

arXiv: https://arxiv.org/abs/2504.02878 · HTML: https://mlsanity.com/paper/2504.02878/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 31 references that are in the map.
- First posted on arXiv 2 Apr 2025.

## Builds on (most cited first, 16 of 16)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [A Random Forest Guided Tour](https://mlsanity.com/paper/1511.05741/) (2015)
- [Deep Metric Learning with Hierarchical Triplet Loss](https://mlsanity.com/paper/1810.06951/) (2018)
- [Penetrative AI: Making LLMs Comprehend the Physical World](https://mlsanity.com/paper/2310.09605/) (2023)
- [LLMSense: Harnessing LLMs for High-level Reasoning Over Spatiotemporal Sensor Traces](https://mlsanity.com/paper/2403.19857/) (2024)
- [SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions](https://mlsanity.com/paper/2412.04036/) (2024)
- [DrHouse: An LLM-empowered Diagnostic Reasoning System through Harnessing Outcomes from Sensor Data and Expert Knowledge](https://mlsanity.com/paper/2405.12541/) (2024)
- [MESEN: Exploit Multimodal Data to Design Unimodal Human Activity Recognition with Few Labels](https://mlsanity.com/paper/2404.01958/) (2024)
- [LLaSA: A Sensor-Aware LLM for Natural Language Reasoning of Human Activity from IMU Data](https://mlsanity.com/paper/2406.14498/) (2024)
- [SensorChat: Answering Qualitative and Quantitative Questions during Long-Term Multimodal Sensor Interactions](https://mlsanity.com/paper/2502.02883/) (2025)
- [Sensor2Text: Enabling Natural Language Interactions for Daily Activity Tracking Using Wearable Sensors](https://mlsanity.com/paper/2410.20034/) (2024)
- [SensorQA: A Question Answering Benchmark for Daily-Life Monitoring](https://mlsanity.com/paper/2501.04974/) (2025)
- [GesturePrint: Enabling User Identification for mmWave-based Gesture Recognition Systems](https://mlsanity.com/paper/2408.05358/) (2024)

## Built on by (most cited first, 5 of 5)

- [On the Feasibility of Using MultiModal LLMs to Execute AR Social Engineering Attacks](https://mlsanity.com/paper/2504.13209/) (2025)
- [Evaluating Visual Prompts with Eye-Tracking Data for MLLM-Based Human Activity Recognition](https://mlsanity.com/paper/2604.09585/) (2026)
- [Agentic Driving Coach: Robustness and Determinism of Agentic AI-Powered Human-in-the-Loop Cyber-Physical Systems](https://mlsanity.com/paper/2604.11705/) (2026)
- [Closing the Modality Gap in Zero-Shot HAR: Contrastive Training and Separability-Optimized Prototypes on IMU Data](https://mlsanity.com/paper/2606.10789/) (2026)
- [EdgeHAR: An Edge-Native Compact Sensor Foundation Model for Human Activity Recognition](https://mlsanity.com/paper/2609.14498/) (2026)

## Abstract (arXiv)

Human activity recognition (HAR) using inertial measurement units (IMUs) increasingly leverages large language models (LLMs), yet existing approaches focus on coarse activities like walking or running. Our preliminary study indicates that pretrained LLMs fail catastrophically on fine-grained HAR tasks such as air-written letter recognition, achieving only near-random guessing accuracy. In this work, we first bridge this gap for flat-surface writing scenarios: by fine-tuning LLMs with a self-collected dataset and few-shot learning, we achieved up to a 129x improvement on 2D data. To extend this to 3D scenarios, we designed an encoder-based pipeline that maps 3D data into 2D equivalents, preserving the spatiotemporal information for robust letter prediction. Our end-to-end pipeline achieves 78% accuracy on word recognition with up to 5 letters in mid-air writing scenarios, establishing LLMs as viable tools for fine-grained HAR.