# Cross-Modal Mutual Learning for Cued Speech Recognition

arXiv: https://arxiv.org/abs/2212.01083 · HTML: https://mlsanity.com/paper/2212.01083/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 19 references that are in the map.
- First posted on arXiv 2 Dec 2022.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [End-to-end Audio-visual Speech Recognition with Conformers](https://mlsanity.com/paper/2102.06657/) (2021)
- [End-to-End Automatic Speech Recognition Integrated With CTC-Based Voice Activity Detection](https://mlsanity.com/paper/2002.00551/) (2020)
- [Re-synchronization using the Hand Preceding Model for Multi-modal Fusion in Automatic Continuous Cued Speech Recognition](https://mlsanity.com/paper/2001.00854/) (2020)
- [A Pilot Study on Mandarin Chinese Cued Speech](https://mlsanity.com/paper/2001.00731/) (2020)
- [Cross-Modal Knowledge Distillation Method for Automatic Cued Speech Recognition](https://mlsanity.com/paper/2106.13686/) (2021)
- [Multistream neural architectures for cued-speech recognition using a pre-trained visual feature extractor and constrained CTC decoding](https://mlsanity.com/paper/2204.04965/) (2022)
- [Investigating the Lombard Effect Influence on End-to-End Audio-Visual Speech Recognition](https://mlsanity.com/paper/1906.02112/) (2019)
- [An Attention Self-supervised Contrastive Learning based Three-stage Model for Hand Shape Feature Representation in Cued Speech](https://mlsanity.com/paper/2106.14016/) (2021)

## Built on by (most cited first, 8 of 8)

- [A Comprehensive Survey on Human Video Generation: Challenges, Methods, and Insights](https://mlsanity.com/paper/2407.08428/) (2024)
- [A Novel Interpretable and Generalizable Re-synchronization Model for Cued Speech based on a Multi-Cuer Corpus](https://mlsanity.com/paper/2306.02596/) (2023)
- [A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation](https://mlsanity.com/paper/2308.08849/) (2023)
- [Computation and Parameter Efficient Multi-Modal Fusion Transformer for Cued Speech Recognition](https://mlsanity.com/paper/2401.17604/) (2024)
- [Bridge to Non-Barrier Communication: Gloss-Prompted Fine-grained Cued Speech Gesture Generation with Diffusion Model](https://mlsanity.com/paper/2404.19277/) (2024)
- [Investigating the dynamics of hand and lips in French Cued Speech using attention mechanisms and CTC-based decoding](https://mlsanity.com/paper/2306.08290/) (2023)
- [UniCUE: Unified Recognition and Generation Framework for Chinese Cued Speech Video-to-Speech Generation](https://mlsanity.com/paper/2506.04134/) (2025)
- [Cued-Agent: A Collaborative Multi-Agent System for Automatic Cued Speech Recognition](https://mlsanity.com/paper/2508.00391/) (2025)

## Abstract (arXiv)

Automatic Cued Speech Recognition (ACSR) provides an intelligent human-machine interface for visual communications, where the Cued Speech (CS) system utilizes lip movements and hand gestures to code spoken language for hearing-impaired people. Previous ACSR approaches often utilize direct feature concatenation as the main fusion paradigm. However, the asynchronous modalities i.e., lip, hand shape and hand position) in CS may cause interference for feature concatenation. To address this challenge, we propose a transformer based cross-modal mutual learning framework to prompt multi-modal interaction. Compared with the vanilla self-attention, our model forces modality-specific information of different modalities to pass through a modality-invariant codebook, collating linguistic representations for tokens of each modality. Then the shared linguistic knowledge is used to re-synchronize multi-modal sequences. Moreover, we establish a novel large-scale multi-speaker CS dataset for Mandarin Chinese. To our knowledge, this is the first work on ACSR for Mandarin Chinese. Extensive experiments are conducted for different languages i.e., Chinese, French, and British English). Results demonstrate that our model exhibits superior recognition performance to the state-of-the-art by a large margin.