# On-device Large Multi-modal Agent for Human Activity Recognition

arXiv: https://arxiv.org/abs/2512.19742 · HTML: https://mlsanity.com/paper/2512.19742/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 18 of its 60 references that are in the map.
- First posted on arXiv 17 Dec 2025.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 18 of 18)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security](https://mlsanity.com/paper/2401.05459/) (2024)
- [Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables](https://mlsanity.com/paper/1604.08880/) (2016)
- [Self-supervised Learning for Human Activity Recognition Using 700,000 Person-days of Wearable Data](https://mlsanity.com/paper/2206.02909/) (2022)
- [Mobile Sensor Data Anonymization](https://mlsanity.com/paper/1810.11546/) (2018)
- [HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?](https://mlsanity.com/paper/2403.02727/) (2024)
- [SensorLLM: Aligning Large Language Models with Motion Sensors for Human Activity Recognition](https://mlsanity.com/paper/2410.10624/) (2024)
- [LLark: A Multimodal Instruction-Following Language Model for Music](https://mlsanity.com/paper/2310.07160/) (2023)
- [Artificial Intelligence of Things: A Survey](https://mlsanity.com/paper/2410.19998/) (2024)
- [Self-Consuming Generative Models with Adversarially Curated Data](https://mlsanity.com/paper/2505.09768/) (2025)
- [Benchmarking Machine Learning: How Fast Can Your Algorithms Go?](https://mlsanity.com/paper/2101.03219/) (2021)
- [ML-based Secure Low-Power Communication in Adversarial Contexts](https://mlsanity.com/paper/2212.13689/) (2022)
- [Design and Implementation Considerations for a Virtual File System Using an Inode Data Structure](https://mlsanity.com/paper/2312.15153/) (2023)
- [Achieving Carbon Neutrality for I/O Devices](https://mlsanity.com/paper/2501.14774/) (2024)
- [A Miniscule Survey on Blockchain Scalability](https://mlsanity.com/paper/2212.13353/) (2022)
- [Analysis of Security in OS-Level Virtualization](https://mlsanity.com/paper/2501.01334/) (2025)

## Built on by (most cited first, 1 of 1)

- [TRACE: Temporal Reasoning over Context and Evidence for Activity Recognition in Smart Homes](https://mlsanity.com/paper/2605.02841/) (2026)

## Abstract (arXiv)

Human Activity Recognition (HAR) has been an active area of research, with applications ranging from healthcare to smart environments. The recent advancements in Large Language Models (LLMs) have opened new possibilities to leverage their capabilities in HAR, enabling not just activity classification but also interpretability and human-like interaction. In this paper, we present a Large Multi-Modal Agent designed for HAR, which integrates the power of LLMs to enhance both performance and user engagement. The proposed framework not only delivers activity classification but also bridges the gap between technical outputs and user-friendly insights through its reasoning and question-answering capabilities. We conduct extensive evaluations using widely adopted HAR datasets, including HHAR, Shoaib, Motionsense to assess the performance of our framework. The results demonstrate that our model achieves high classification accuracy comparable to state-of-the-art methods while significantly improving interpretability through its reasoning and Q&A capabilities.