# An Attention Self-supervised Contrastive Learning based Three-stage Model for Hand Shape Feature Representation in Cued Speech

arXiv: https://arxiv.org/abs/2106.14016 · HTML: https://mlsanity.com/paper/2106.14016/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jun 2021.

## Built on by (most cited first, 4 of 4)

- [Cross-Modal Mutual Learning for Cued Speech Recognition](https://mlsanity.com/paper/2212.01083/) (2022)
- [A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation](https://mlsanity.com/paper/2308.08849/) (2023)
- [Computation and Parameter Efficient Multi-Modal Fusion Transformer for Cued Speech Recognition](https://mlsanity.com/paper/2401.17604/) (2024)
- [Cued-Agent: A Collaborative Multi-Agent System for Automatic Cued Speech Recognition](https://mlsanity.com/paper/2508.00391/) (2025)

## Abstract (arXiv)

Cued Speech (CS) is a communication system for deaf people or hearing impaired people, in which a speaker uses it to aid a lipreader in phonetic level by clarifying potentially ambiguous mouth movements with hand shape and positions. Feature extraction of multi-modal CS is a key step in CS recognition. Recent supervised deep learning based methods suffer from noisy CS data annotations especially for hand shape modality. In this work, we first propose a self-supervised contrastive learning method to learn the feature representation of image without using labels. Secondly, a small amount of manually annotated CS data are used to fine-tune the first module. Thirdly, we present a module, which combines Bi-LSTM and self-attention networks to further learn sequential features with temporal and contextual information. Besides, to enlarge the volume and the diversity of the current limited CS datasets, we build a new British English dataset containing 5 native CS speakers. Evaluation results on both French and British English datasets show that our model achieves over 90% accuracy in hand shape recognition. Significant improvements of 8.75% (for French) and 10.09% (for British English) are achieved in CS phoneme recognition correctness compared with the state-of-the-art.