# Concurrent Activity Recognition with Multimodal CNN-LSTM Structure

arXiv: https://arxiv.org/abs/1702.01638 · HTML: https://mlsanity.com/paper/1702.01638/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Feb 2017.

## Built on by (most cited first, 5 of 5)

- [A Survey of Human Activity Recognition in Smart Homes Based on IoT Sensors Algorithms: Taxonomies, Challenges, and Opportunities with Deep Learning](https://mlsanity.com/paper/2111.04418/) (2021)
- [H2H: Heterogeneous Model to Heterogeneous System Mapping with Computation and Communication Awareness](https://mlsanity.com/paper/2204.13852/) (2022)
- [Research on Data Fusion Algorithm Based on Deep Learning in Target Tracking](https://mlsanity.com/paper/2211.12776/) (2022)
- [Towards Multi-User Activity Recognition through Facilitated Training Data and Deep Learning for Human-Robot Collaboration Applications](https://mlsanity.com/paper/2302.05763/) (2023)
- [Game Theory Solutions in Sensor-Based Human Activity Recognition: A Review](https://mlsanity.com/paper/2311.06311/) (2023)

## Abstract (arXiv)

We introduce a system that recognizes concurrent activities from real-world data captured by multiple sensors of different types. The recognition is achieved in two steps. First, we extract spatial and temporal features from the multimodal data. We feed each datatype into a convolutional neural network that extracts spatial features, followed by a long-short term memory network that extracts temporal information in the sensory data. The extracted features are then fused for decision making in the second step. Second, we achieve concurrent activity recognition with a single classifier that encodes a binary output vector in which elements indicate whether the corresponding activity types are currently in progress. We tested our system with three datasets from different domains recorded using different sensors and achieved performance comparable to existing systems designed specifically for those domains. Our system is the first to address the concurrent activity recognition with multisensory data using a single model, which is scalable, simple to train and easy to deploy.