# Large Language Model-Guided Semantic Alignment for Human Activity Recognition

arXiv: https://arxiv.org/abs/2410.00003 · HTML: https://mlsanity.com/paper/2410.00003/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 57 references that are in the map.
- First posted on arXiv 12 Sep 2024.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [AutoDroid: LLM-powered Task Automation in Android](https://mlsanity.com/paper/2308.15272/) (2023)
- [Data Augmentation of Wearable Sensor Data for Parkinson's Disease Monitoring using Convolutional Neural Networks](https://mlsanity.com/paper/1706.00527/) (2017)
- [Multi-task Self-Supervised Learning for Human Activity Detection](https://mlsanity.com/paper/1907.11879/) (2019)
- [DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing](https://mlsanity.com/paper/1611.01942/) (2016)
- [Mobile Sensor Data Anonymization](https://mlsanity.com/paper/1810.11546/) (2018)
- [Penetrative AI: Making LLMs Comprehend the Physical World](https://mlsanity.com/paper/2310.09605/) (2023)
- [Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automation](https://mlsanity.com/paper/2312.03003/) (2023)
- [HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?](https://mlsanity.com/paper/2403.02727/) (2024)
- [What Makes Good Contrastive Learning on Small-Scale Wearable-based Tasks?](https://mlsanity.com/paper/2202.05998/) (2022)
- [SelfHAR: Improving Human Activity Recognition through Self-training with Unlabeled Data](https://mlsanity.com/paper/2102.06073/) (2021)
- [Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognition](https://mlsanity.com/paper/2206.06629/) (2022)
- [Combining Public Human Activity Recognition Datasets to Mitigate Labeled Data Scarcity](https://mlsanity.com/paper/2306.13735/) (2023)
- [Using Large Language Models to Compare Explainable Models for Smart Home Human Activity Recognition](https://mlsanity.com/paper/2408.06352/) (2024)
- [Unleashing the Power of Shared Label Structures for Human Activity Recognition](https://mlsanity.com/paper/2301.03462/) (2023)

## Built on by (most cited first, 6 of 6)

- [Mojito: LLM-Aided Motion Instructor with Jitter-Reduced Inertial Tokens](https://mlsanity.com/paper/2502.16175/) (2025)
- [BenchHAR: Benchmarking Self-Supervised Learning for Generalizable Sensor-based Activity Recognition](https://mlsanity.com/paper/2605.08296/) (2026)
- [Evaluating Visual Prompts with Eye-Tracking Data for MLLM-Based Human Activity Recognition](https://mlsanity.com/paper/2604.09585/) (2026)
- [TRACE: Temporal Reasoning over Context and Evidence for Activity Recognition in Smart Homes](https://mlsanity.com/paper/2605.02841/) (2026)
- [Closing the Modality Gap in Zero-Shot HAR: Contrastive Training and Separability-Optimized Prototypes on IMU Data](https://mlsanity.com/paper/2606.10789/) (2026)
- [OntoAR-HAR: Ontology-Guided Agentic RAG with Personalized Memory for Human Activity Recognition](https://mlsanity.com/paper/2610.15292/) (2026)

## Abstract (arXiv)

Human Activity Recognition (HAR) using Inertial Measurement Unit (IMU) sensors is critical for applications in healthcare, safety, and industrial production. However, variations in activity patterns, device types, and sensor placements create distribution gaps across datasets, reducing the performance of HAR models. To address this, we propose LanHAR, a novel system that leverages Large Language Models (LLMs) to generate semantic interpretations of sensor readings and activity labels for cross-dataset HAR. This approach not only mitigates cross-dataset heterogeneity but also enhances the recognition of new activities. LanHAR employs an iterative re-generation method to produce high-quality semantic interpretations with LLMs and a two-stage training framework that bridges the semantic interpretations of sensor readings and activity labels. This ultimately leads to a lightweight sensor encoder suitable for mobile deployment, enabling any sensor reading to be mapped into the semantic interpretation space. Experiments on five public datasets demonstrate that our approach significantly outperforms state-of-the-art methods in both cross-dataset HAR and new activity recognition. The source code is publicly available at https://github.com/DASHLab/LanHAR.