# A Survey on Multimodal Wearable Sensor-based Human Action Recognition arXiv: https://arxiv.org/abs/2404.15349 ยท HTML: https://mlsanity.com/paper/2404.15349/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 123 of its 274 references that are in the map. - First posted on arXiv 14 Apr 2024. ## Builds on (most cited first, 30 of 123) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022) - [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017) - [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023) - [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022) - [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019) - [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018) - [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022) - [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021) - [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018) - [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023) - [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014) - [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018) - [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023) - [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018) - [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023) - [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019) - [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020) - [End-to-End Learning of Visual Representations from Uncurated Instructional Videos](https://mlsanity.com/paper/1912.06430/) (2019) - [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015) - [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021) - [Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks](https://mlsanity.com/paper/2206.08916/) (2022) - [U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation](https://mlsanity.com/paper/2401.04722/) (2024) - [Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers](https://mlsanity.com/paper/2402.19479/) (2024) - [A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT](https://mlsanity.com/paper/2302.09419/) (2023) - [VideoMamba: State Space Model for Efficient Video Understanding](https://mlsanity.com/paper/2403.06977/) (2024) - [VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks](https://mlsanity.com/paper/2112.06825/) (2021) - [MM-LLMs: Recent Advances in MultiModal Large Language Models](https://mlsanity.com/paper/2401.13601/) (2024) - [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020) ## Built on by (most cited first, 10 of 10) - [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024) - [Towards Generalizable Human Activity Recognition: A Survey](https://mlsanity.com/paper/2508.12213/) (2025) - [ComBodied Agents: a New Paradigm of Human-Centric Agentic AI](https://mlsanity.com/paper/2608.10915/) (2026) - [HERMES: A Unified Open-Source Framework for Realtime Multimodal Physiological Sensing, Edge AI, and Intervention in Closed-Loop Smart Healthcare Applications](https://mlsanity.com/paper/2601.12610/) (2026) - [TxP: Reciprocal Generation of Ground Pressure Dynamics and Activity Descriptions for Improving Human Activity Recognition](https://mlsanity.com/paper/2505.02052/) (2025) - [Non-stationary BERT: Exploring Augmented IMU Data For Robust Human Activity Recognition](https://mlsanity.com/paper/2409.16730/) (2024) - [Ground Reaction Force Estimation via Time-aware Knowledge Distillation](https://mlsanity.com/paper/2506.10265/) (2025) - [Privacy-Preserving Multi-Stage Fall Detection Framework with Semi-supervised Federated Learning and Robotic Vision Confirmation](https://mlsanity.com/paper/2507.10474/) (2025) - [DETACH : Decomposed Spatio-Temporal Alignment for Exocentric Video and Ambient Sensors with Staged Learning](https://mlsanity.com/paper/2512.20409/) (2025) - [Selective Correlation Based Knowledge Distillation for Ground Reaction Force Estimation](https://mlsanity.com/paper/2605.00888/) (2026) ## Abstract (arXiv) The combination of increased life expectancy and falling birth rates is resulting in an aging population. Wearable Sensor-based Human Activity Recognition (WSHAR) emerges as a promising assistive technology to support the daily lives of older individuals, unlocking vast potential for human-centric applications. However, recent surveys in WSHAR have been limited, focusing either solely on deep learning approaches or on a single sensor modality. In real life, our human interact with the world in a multi-sensory way, where diverse information sources are intricately processed and interpreted to accomplish a complex and unified sensing system. To give machines similar intelligence, multimodal machine learning, which merges data from various sources, has become a popular research area with recent advancements. In this study, we present a comprehensive survey from a novel perspective on how to leverage multimodal learning to WSHAR domain for newcomers and researchers. We begin by presenting the recent sensor modalities as well as deep learning approaches in HAR. Subsequently, we explore the techniques used in present multimodal systems for WSHAR. This includes inter-multimodal systems which utilize sensor modalities from both visual and non-visual systems and intra-multimodal systems that simply take modalities from non-visual systems. After that, we focus on current multimodal learning approaches that have applied to solve some of the challenges existing in WSHAR. Specifically, we make extra efforts by connecting the existing multimodal literature from other domains, such as computer vision and natural language processing, with current WSHAR area. Finally, we identify the corresponding challenges and potential research direction in current WSHAR area for further improvement.