# Towards Improved Human Action Recognition Using Convolutional Neural Networks and Multimodal Fusion of Depth and Inertial Sensor Data

arXiv: https://arxiv.org/abs/2008.09747 · HTML: https://mlsanity.com/paper/2008.09747/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2020.

## Built on by (most cited first, 4 of 4)

- [ECG Heartbeat Classification Using Multimodal Fusion](https://mlsanity.com/paper/2107.09869/) (2021)
- [Multi-level Stress Assessment from ECG in a Virtual Reality Environment using Multimodal Fusion](https://mlsanity.com/paper/2107.04566/) (2021)
- [Inertial Sensor Data To Image Encoding For Human Action Recognition](https://mlsanity.com/paper/2105.13533/) (2021)
- [A Survey on Human Action Recognition](https://mlsanity.com/paper/2301.06082/) (2022)

## Abstract (arXiv)

This paper attempts at improving the accuracy of Human Action Recognition (HAR) by fusion of depth and inertial sensor data. Firstly, we transform the depth data into Sequential Front view Images(SFI) and fine-tune the pre-trained AlexNet on these images. Then, inertial data is converted into Signal Images (SI) and another convolutional neural network (CNN) is trained on these images. Finally, learned features are extracted from both CNN, fused together to make a shared feature layer, and these features are fed to the classifier. We experiment with two classifiers, namely Support Vector Machines (SVM) and softmax classifier and compare their performances. The recognition accuracies of each modality, depth data alone and sensor data alone are also calculated and compared with fusion based accuracies to highlight the fact that fusion of modalities yields better results than individual modalities. Experimental results on UTD-MHAD and Kinect 2D datasets show that proposed method achieves state of the art results when compared to other recently proposed visual-inertial action recognition methods.