# Multi-Modality Fusion based on Consensus-Voting and 3D Convolution for Isolated Gesture Recognition

arXiv: https://arxiv.org/abs/1611.06689 · HTML: https://mlsanity.com/paper/1611.06689/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2016.

## Built on by (most cited first, 3 of 3)

- [A Comprehensive Review of Sign Language Recognition: Different Types, Modalities, and Datasets](https://mlsanity.com/paper/2204.03328/) (2022)
- [Bridging Gap between Image Pixels and Semantics via Supervision: A Survey](https://mlsanity.com/paper/2107.13757/) (2021)
- [Transformers in Action Recognition: A Review on Temporal Modeling](https://mlsanity.com/paper/2302.01921/) (2022)

## Abstract (arXiv)

Recently, the popularity of depth-sensors such as Kinect has made depth videos easily available while its advantages have not been fully exploited. This paper investigates, for gesture recognition, to explore the spatial and temporal information complementarily embedded in RGB and depth sequences. We propose a convolutional twostream consensus voting network (2SCVN) which explicitly models both the short-term and long-term structure of the RGB sequences. To alleviate distractions from background, a 3d depth-saliency ConvNet stream (3DDSN) is aggregated in parallel to identify subtle motion characteristics. These two components in an unified framework significantly improve the recognition accuracy. On the challenging Chalearn IsoGD benchmark, our proposed method outperforms the first place on the leader-board by a large margin (10.29%) while also achieving the best result on RGBD-HuDaAct dataset (96.74%). Both quantitative experiments and qualitative analysis shows the effectiveness of our proposed framework and codes will be released to facilitate future research.