# CNN based Multistage Gated Average Fusion (MGAF) for Human Action Recognition Using Depth and Inertial Sensors

arXiv: https://arxiv.org/abs/2010.16073 · HTML: https://mlsanity.com/paper/2010.16073/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2020.

## Built on by (most cited first, 10 of 10)

- [ECG Heartbeat Classification Using Multimodal Fusion](https://mlsanity.com/paper/2107.09869/) (2021)
- [Progressive Cross-modal Knowledge Distillation for Human Action Recognition](https://mlsanity.com/paper/2208.08090/) (2022)
- [ModSelect: Automatic Modality Selection for Synthetic-to-Real Domain Generalization](https://mlsanity.com/paper/2208.09414/) (2022)
- [TransFusionOdom: Interpretable Transformer-based LiDAR-Inertial Fusion Odometry Estimation](https://mlsanity.com/paper/2304.07728/) (2023)
- [Multi-level Stress Assessment from ECG in a Virtual Reality Environment using Multimodal Fusion](https://mlsanity.com/paper/2107.04566/) (2021)
- [ChMusic: A Traditional Chinese Music Dataset for Evaluation of Instrument Recognition](https://mlsanity.com/paper/2108.08470/) (2021)
- [Inertial Sensor Data To Image Encoding For Human Action Recognition](https://mlsanity.com/paper/2105.13533/) (2021)
- [A Novel Two Stream Decision Level Fusion of Vision and Inertial Sensors Data for Automatic Multimodal Human Activity Recognition System](https://mlsanity.com/paper/2306.15765/) (2023)
- [DEAR: Depth-Enhanced Action Recognition](https://mlsanity.com/paper/2408.15679/) (2024)
- [A Survey on Human Action Recognition](https://mlsanity.com/paper/2301.06082/) (2022)

## Abstract (arXiv)

Convolutional Neural Network (CNN) provides leverage to extract and fuse features from all layers of its architecture. However, extracting and fusing intermediate features from different layers of CNN structure is still uninvestigated for Human Action Recognition (HAR) using depth and inertial sensors. To get maximum benefit of accessing all the CNN's layers, in this paper, we propose novel Multistage Gated Average Fusion (MGAF) network which extracts and fuses features from all layers of CNN using our novel and computationally efficient Gated Average Fusion (GAF) network, a decisive integral element of MGAF. At the input of the proposed MGAF, we transform the depth and inertial sensor data into depth images called sequential front view images (SFI) and signal images (SI) respectively. These SFI are formed from the front view information generated by depth data. CNN is employed to extract feature maps from both input modalities. GAF network fuses the extracted features effectively while preserving the dimensionality of fused feature as well. The proposed MGAF network has structural extensibility and can be unfolded to more than two modalities. Experiments on three publicly available multimodal HAR datasets demonstrate that the proposed MGAF outperforms the previous state of the art fusion methods for depth-inertial HAR in terms of recognition accuracy while being computationally much more efficient. We increase the accuracy by an average of 1.5 percent while reducing the computational cost by approximately 50 percent over the previous state of the art.