# ActivityNarrated: An Open-Ended Narrative Paradigm for Wearable Human Activity Understanding

arXiv: https://arxiv.org/abs/2604.00767 · HTML: https://mlsanity.com/paper/2604.00767/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 38 references that are in the map.
- First posted on arXiv 1 Apr 2026.

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively](https://mlsanity.com/paper/2401.02955/) (2024)
- [Assessing the State of Self-Supervised Human Activity Recognition using Wearables](https://mlsanity.com/paper/2202.12938/) (2022)
- [Ensembles of Deep LSTM Learners for Activity Recognition using Wearables](https://mlsanity.com/paper/1703.09370/) (2017)
- [Contrastive Predictive Coding for Human Activity Recognition](https://mlsanity.com/paper/2012.05333/) (2020)
- [Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimization](https://mlsanity.com/paper/2302.00624/) (2023)
- [HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?](https://mlsanity.com/paper/2403.02727/) (2024)
- [SensorLM: Learning the Language of Wearable Sensors](https://mlsanity.com/paper/2506.09108/) (2025)
- [On Attention Models for Human Activity Recognition](https://mlsanity.com/paper/1805.07648/) (2018)
- [WEAR: An Outdoor Sports Dataset for Wearable and Egocentric Activity Recognition](https://mlsanity.com/paper/2304.05088/) (2023)
- [Initial Findings on Sensor based Open Vocabulary Activity Recognition via Text Embedding Inversion](https://mlsanity.com/paper/2501.07408/) (2025)
- [OpenMarcie: Dataset for Multimodal Action Recognition in Industrial Environments](https://mlsanity.com/paper/2603.02390/) (2026)

## Abstract (arXiv)

Wearable human activity recognition (HAR) has made steady progress, yet much of this progress remains grounded in fixed-window, closed-set classification benchmarks. This formulation is poorly matched to everyday behavior, where activities are open-ended, unscripted, personalized, variable in duration, and often compositional. To address this mismatch, we introduce ActivityNarrated, an open-ended narrative paradigm for language-grounded wearable activity understanding. We formulate this setting as dense sensor signal captioning with a comprehensive benchmark protocol that measures temporal localization, caption quality, sensor-language alignment, conventional closed-set classification as a downstream diagnostic, and additional robustness measures. We further present ActNarrator, a 3-stage architecture that discretizes continuous IMU signals into reusable motion tokens and uses an external frozen small language model to generate open-vocabulary activity captions. Experiments show that our method provides high quality dense sensor captioning with superior adaptivity and robustness, enabling various downstream tasks by turning sensor-based human activity understanding into sensor-grounded text-level reasoning. This includes downstream classification where ActNarrator outperforms state-of-the-art HAR models by 3.8 - 31.6 % in Macro-F1. This paradigm also enables novel activity understanding capabilities such as complex question-answering over long time horizons.