# Fusion-GCN: Multimodal Action Recognition using Graph Convolutional Networks

arXiv: https://arxiv.org/abs/2109.12946 · HTML: https://mlsanity.com/paper/2109.12946/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 55 references that are in the map.
- First posted on arXiv 27 Sep 2021.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [Videos as Space-Time Region Graphs](https://mlsanity.com/paper/1806.01810/) (2018)
- [MMTM: Multimodal Transfer Module for CNN Fusion](https://mlsanity.com/paper/1911.08670/) (2019)
- [PifPaf: Composite Fields for Human Pose Estimation](https://mlsanity.com/paper/1903.06593/) (2019)
- [Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition](https://mlsanity.com/paper/2010.09978/) (2020)
- [MFAS: Multimodal Fusion Architecture Search](https://mlsanity.com/paper/1903.06496/) (2019)
- [Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching](https://mlsanity.com/paper/1911.04131/) (2019)
- [XNect: Real-time Multi-Person 3D Motion Capture with a Single RGB Camera](https://mlsanity.com/paper/1907.00837/) (2019)
- [P-CNN: Pose-based CNN Features for Action Recognition](https://mlsanity.com/paper/1506.03607/) (2015)
- [Learning Conditioned Graph Structures for Interpretable Visual Question Answering](https://mlsanity.com/paper/1806.07243/) (2018)
- [Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition](https://mlsanity.com/paper/2009.00210/) (2020)
- [Graph Distillation for Action Detection with Privileged Modalities](https://mlsanity.com/paper/1712.00108/) (2017)
- [JOLO-GCN: Mining Joint-Centered Light-Weight Information for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2011.07787/) (2020)
- [HAMLET: A Hierarchical Multimodal Attention-based Human Activity Recognition Algorithm](https://mlsanity.com/paper/2008.01148/) (2020)
- [Gimme Signals: Discriminative signal encoding for multimodal activity recognition](https://mlsanity.com/paper/2003.06156/) (2020)
- [Body Joint guided 3D Deep Convolutional Descriptors for Action Recognition](https://mlsanity.com/paper/1704.07160/) (2017)
- [Vertex Feature Encoding and Hierarchical Temporal Modeling in a Spatial-Temporal Graph Convolutional Network for Action Recognition](https://mlsanity.com/paper/1912.09745/) (2019)
- [A Dual-Source Approach for 3D Human Pose Estimation from a Single Image](https://mlsanity.com/paper/1705.02883/) (2017)
- [Vision-Based Fallen Person Detection for the Elderly](https://mlsanity.com/paper/1707.07608/) (2017)
- [HAUAR: Home Automation Using Action Recognition](https://mlsanity.com/paper/1904.10354/) (2019)

## Built on by (most cited first, 15 of 15)

- [End-to-End Active Speaker Detection](https://mlsanity.com/paper/2203.14250/) (2022)
- [Progressive Cross-modal Knowledge Distillation for Human Action Recognition](https://mlsanity.com/paper/2208.08090/) (2022)
- [MuJo: Multimodal Joint Feature Space Learning for Human Activity Recognition](https://mlsanity.com/paper/2406.03857/) (2024)
- [A Non-Anatomical Graph Structure for isolated hand gesture separation in continuous gesture sequences](https://mlsanity.com/paper/2207.07619/) (2022)
- [VS-CAM: Vertex Semantic Class Activation Mapping to Interpret Vision Graph Neural Network](https://mlsanity.com/paper/2209.09104/) (2022)
- [A Network Science perspective of Graph Convolutional Networks: A survey](https://mlsanity.com/paper/2301.04824/) (2023)
- [Component attention network for multimodal dance improvisation recognition](https://mlsanity.com/paper/2310.05938/) (2023)
- [Evolving Skeletons: Motion Dynamics in Action Recognition](https://mlsanity.com/paper/2501.02593/) (2025)
- [Leveraging Foundation Models for Multimodal Graph-Based Action Recognition](https://mlsanity.com/paper/2505.15192/) (2025)
- [Unified Contrastive Fusion Transformer for Multimodal Human Action Recognition](https://mlsanity.com/paper/2309.05032/) (2023)
- [Modality Mixer Exploiting Complementary Information for Multi-modal Action Recognition](https://mlsanity.com/paper/2311.12344/) (2023)
- [Improving Skeleton-based Action Recognition with Interactive Object Information](https://mlsanity.com/paper/2501.05066/) (2025)
- [Multimodal Sheaf-based Network for Glioblastoma Molecular Subtype Prediction](https://mlsanity.com/paper/2508.09717/) (2025)
- [Why Relational Graphs Will Save the Next Generation of Vision Foundation Models?](https://mlsanity.com/paper/2508.18421/) (2025)
- [Recognition of Daily Activities through Multi-Modal Deep Learning: A Video, Pose, and Object-Aware Approach for Ambient Assisted Living](https://mlsanity.com/paper/2603.04509/) (2026)

## Abstract (arXiv)

In this paper, we present Fusion-GCN, an approach for multimodal action recognition using Graph Convolutional Networks (GCNs). Action recognition methods based around GCNs recently yielded state-of-the-art performance for skeleton-based action recognition. With Fusion-GCN, we propose to integrate various sensor data modalities into a graph that is trained using a GCN model for multi-modal action recognition. Additional sensor measurements are incorporated into the graph representation, either on a channel dimension (introducing additional node attributes) or spatial dimension (introducing new nodes). Fusion-GCN was evaluated on two public available datasets, the UTD-MHAD- and MMACT datasets, and demonstrates flexible fusion of RGB sequences, inertial measurements and skeleton sequences. Our approach gets comparable results on the UTD-MHAD dataset and improves the baseline on the large-scale MMACT dataset by a significant margin of up to 12.37% (F1-Measure) with the fusion of skeleton estimates and accelerometer measurements.