# A Novel Two Stream Decision Level Fusion of Vision and Inertial Sensors Data for Automatic Multimodal Human Activity Recognition System

arXiv: https://arxiv.org/abs/2306.15765 · HTML: https://mlsanity.com/paper/2306.15765/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 54 references that are in the map.
- First posted on arXiv 27 Jun 2023.

## Builds on (most cited first, 13 of 13)

- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation](https://mlsanity.com/paper/1908.10357/) (2019)
- [Deep Learning for Sensor-based Activity Recognition: A Survey](https://mlsanity.com/paper/1707.03502/) (2017)
- [Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition](https://mlsanity.com/paper/2009.00210/) (2020)
- [Gimme Signals: Discriminative signal encoding for multimodal activity recognition](https://mlsanity.com/paper/2003.06156/) (2020)
- [CNN based Multistage Gated Average Fusion (MGAF) for Human Action Recognition Using Depth and Inertial Sensors](https://mlsanity.com/paper/2010.16073/) (2020)
- [Action Machine: Rethinking Action Recognition in Trimmed Videos](https://mlsanity.com/paper/1812.05770/) (2018)
- [Human Action Recognition Using Deep Multilevel Multimodal (M2) Fusion of Depth and Inertial Sensors](https://mlsanity.com/paper/1910.11482/) (2019)
- [Real-Time Human Motion Capture with Multiple Depth Cameras](https://mlsanity.com/paper/1605.08068/) (2016)
- [Robust Automated Human Activity Recognition and its Application to Sleep Research](https://mlsanity.com/paper/1607.04867/) (2016)
- [Fusing Video and Inertial Sensor Data for Walking Person Identification](https://mlsanity.com/paper/1802.07021/) (2018)

## Built on by (most cited first, 1 of 1)

- [X-Fi: A Modality-Invariant Foundation Model for Multimodal Human Sensing](https://mlsanity.com/paper/2410.10167/) (2024)

## Abstract (arXiv)

This paper presents a novel multimodal human activity recognition system. It uses a two-stream decision level fusion of vision and inertial sensors. In the first stream, raw RGB frames are passed to a part affinity field-based pose estimation network to detect the keypoints of the user. These keypoints are then pre-processed and inputted in a sliding window fashion to a specially designed convolutional neural network for the spatial feature extraction followed by regularized LSTMs to calculate the temporal features. The outputs of LSTM networks are then inputted to fully connected layers for classification. In the second stream, data obtained from inertial sensors are pre-processed and inputted to regularized LSTMs for the feature extraction followed by fully connected layers for the classification. At this stage, the SoftMax scores of two streams are then fused using the decision level fusion which gives the final prediction. Extensive experiments are conducted to evaluate the performance. Four multimodal standard benchmark datasets (UP-Fall detection, UTD-MHAD, Berkeley-MHAD, and C-MHAD) are used for experimentations. The accuracies obtained by the proposed system are 96.9 %, 97.6 %, 98.7 %, and 95.9 % respectively on the UP-Fall Detection, UTDMHAD, Berkeley-MHAD, and C-MHAD datasets. These results are far superior than the current state-of-the-art methods.