# Multi-modal Fusion for Single-Stage Continuous Gesture Recognition

arXiv: https://arxiv.org/abs/2011.04945 · HTML: https://mlsanity.com/paper/2011.04945/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2020.

## Built on by (most cited first, 10 of 10)

- [Continuous Human Action Recognition for Human-Machine Interaction: A Review](https://mlsanity.com/paper/2202.13096/) (2022)
- [Timestamp-supervised Wearable-based Activity Segmentation and Recognition with Contrastive Learning and Order-Preserving Optimal Transport](https://mlsanity.com/paper/2310.09114/) (2023)
- [StepNet: Spatial-temporal Part-aware Network for Isolated Sign Language Recognition](https://mlsanity.com/paper/2212.12857/) (2022)
- [Vision-Based Activity Recognition in Children with Autism-Related Behaviors](https://mlsanity.com/paper/2208.04206/) (2022)
- [Multi-level Contrast Network for Wearables-based Joint Activity Segmentation and Recognition](https://mlsanity.com/paper/2208.07547/) (2022)
- [Real-Time Hand Gesture Recognition: Integrating Skeleton-Based Data Fusion and Multi-Stream CNN](https://mlsanity.com/paper/2406.15003/) (2024)
- [Multiscaled Multi-Head Attention-based Video Transformer Network for Hand Gesture Recognition](https://mlsanity.com/paper/2501.00935/) (2025)
- [mmEgoHand: Egocentric Hand Pose Estimation and Gesture Recognition with Head-mounted Millimeter-wave Radar and IMU](https://mlsanity.com/paper/2501.13805/) (2025)
- [Robust Multimodal Fusion for Human Activity Recognition](https://mlsanity.com/paper/2303.04636/) (2023)
- [Dual-Stream Alignment for Action Segmentation](https://mlsanity.com/paper/2510.07652/) (2025)

## Abstract (arXiv)

Gesture recognition is a much studied research area which has myriad real-world applications including robotics and human-machine interaction. Current gesture recognition methods have focused on recognising isolated gestures, and existing continuous gesture recognition methods are limited to two-stage approaches where independent models are required for detection and classification, with the performance of the latter being constrained by detection performance. In contrast, we introduce a single-stage continuous gesture recognition framework, called Temporal Multi-Modal Fusion (TMMF), that can detect and classify multiple gestures in a video via a single model. This approach learns the natural transitions between gestures and non-gestures without the need for a pre-processing segmentation step to detect individual gestures. To achieve this, we introduce a multi-modal fusion mechanism to support the integration of important information that flows from multi-modal inputs, and is scalable to any number of modes. Additionally, we propose Unimodal Feature Mapping (UFM) and Multi-modal Feature Mapping (MFM) models to map uni-modal features and the fused multi-modal features respectively. To further enhance performance, we propose a mid-point based loss function that encourages smooth alignment between the ground truth and the prediction, helping the model to learn natural gesture transitions. We demonstrate the utility of our proposed framework, which can handle variable-length input videos, and outperforms the state-of-the-art on three challenging datasets: EgoGesture, IPN hand, and ChaLearn LAP Continuous Gesture Dataset (ConGD). Furthermore, ablation experiments show the importance of different components of the proposed framework.