# Dual-view Spatio-Temporal Feature Fusion with CNN-Transformer Hybrid Network for Chinese Isolated Sign Language Recognition

arXiv: https://arxiv.org/abs/2506.06966 · HTML: https://mlsanity.com/paper/2506.06966/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 39 references that are in the map.
- First posted on arXiv 8 Jun 2025.

## Builds on (most cited first, 19 of 19)

- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison](https://mlsanity.com/paper/1910.11006/) (2019)
- [How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language](https://mlsanity.com/paper/2008.08143/) (2020)
- [An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1611.06067/) (2016)
- [MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language](https://mlsanity.com/paper/1812.01053/) (2018)
- [Skeleton Aware Multi-modal Sign Language Recognition](https://mlsanity.com/paper/2103.08833/) (2021)
- [AUTSL: A Large Scale Multi-modal Turkish Sign Language Dataset and Baseline Methods](https://mlsanity.com/paper/2008.00932/) (2020)
- [SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition](https://mlsanity.com/paper/2110.05382/) (2021)
- [SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2305.04868/) (2023)
- [VPN: Learning Video-Pose Embedding for Activities of Daily Living](https://mlsanity.com/paper/2007.03056/) (2020)
- [Natural Language-Assisted Sign Language Recognition](https://mlsanity.com/paper/2303.12080/) (2023)
- [Global-local Enhancement Network for NMFs-aware Sign Language Recognition](https://mlsanity.com/paper/2008.10428/) (2020)
- [Unsupervised Learning of View-invariant Action Representations](https://mlsanity.com/paper/1809.01844/) (2018)
- [Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment](https://mlsanity.com/paper/2312.15645/) (2023)
- [AdaSGN: Adapting Joint Number and Model Size for Efficient Skeleton-Based Action Recognition](https://mlsanity.com/paper/2103.11770/) (2021)
- [ViewCLR: Learning Self-supervised Video Representation for Unseen Viewpoints](https://mlsanity.com/paper/2112.03905/) (2021)
- [MM-WLAuslan: Multi-View Multi-Modal Word-Level Australian Sign Language Recognition Dataset](https://mlsanity.com/paper/2410.19488/) (2024)
- [M2DAR: Multi-View Multi-Scale Driver Action Recognition with Vision Transformer](https://mlsanity.com/paper/2305.08877/) (2023)
- [Transfer Learning for Cross-dataset Isolated Sign Language Recognition in Under-Resourced Datasets](https://mlsanity.com/paper/2403.14534/) (2024)

## Built on by (most cited first, 2 of 2)

- [Towards Multi-View Sign Language Understanding: A Benchmark Dataset and Baseline](https://mlsanity.com/paper/2604.18184/) (2026)
- [OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder](https://mlsanity.com/paper/2605.01506/) (2026)

## Abstract (arXiv)

Due to the emergence of many sign language datasets, isolated sign language recognition (ISLR) has made significant progress in recent years. In addition, the development of various advanced deep neural networks is another reason for this breakthrough. However, challenges remain in applying the technique in the real world. First, existing sign language datasets do not cover the whole sign vocabulary. Second, most of the sign language datasets provide only single view RGB videos, which makes it difficult to handle hand occlusions when performing ISLR. To fill this gap, this paper presents a dual-view sign language dataset for ISLR named NationalCSL-DP, which fully covers the Chinese national sign language vocabulary. The dataset consists of 134140 sign videos recorded by ten signers with respect to two vertical views, namely, the front side and the left side. Furthermore, a CNN transformer network is also proposed as a strong baseline and an extremely simple but effective fusion strategy for prediction. Extensive experiments were conducted to prove the effectiveness of the datasets as well as the baseline. The results show that the proposed fusion strategy can significantly increase the performance of the ISLR, but it is not easy for the sequence-to-sequence model, regardless of whether the early-fusion or late-fusion strategy is applied, to learn the complementary features from the sign videos of two vertical views.