# A Novel Interpretable and Generalizable Re-synchronization Model for Cued Speech based on a Multi-Cuer Corpus

arXiv: https://arxiv.org/abs/2306.02596 · HTML: https://mlsanity.com/paper/2306.02596/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 29 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Builds on (most cited first, 7 of 7)

- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Re-synchronization using the Hand Preceding Model for Multi-modal Fusion in Automatic Continuous Cued Speech Recognition](https://mlsanity.com/paper/2001.00854/) (2020)
- [A Pilot Study on Mandarin Chinese Cued Speech](https://mlsanity.com/paper/2001.00731/) (2020)
- [Cross-Modal Knowledge Distillation Method for Automatic Cued Speech Recognition](https://mlsanity.com/paper/2106.13686/) (2021)
- [Cross-Modal Mutual Learning for Cued Speech Recognition](https://mlsanity.com/paper/2212.01083/) (2022)
- [Multistream neural architectures for cued-speech recognition using a pre-trained visual feature extractor and constrained CTC decoding](https://mlsanity.com/paper/2204.04965/) (2022)

## Built on by (most cited first, 3 of 3)

- [A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation](https://mlsanity.com/paper/2308.08849/) (2023)
- [Computation and Parameter Efficient Multi-Modal Fusion Transformer for Cued Speech Recognition](https://mlsanity.com/paper/2401.17604/) (2024)
- [Cued-Agent: A Collaborative Multi-Agent System for Automatic Cued Speech Recognition](https://mlsanity.com/paper/2508.00391/) (2025)

## Abstract (arXiv)

Cued Speech (CS) is a multi-modal visual coding system combining lip reading with several hand cues at the phonetic level to make the spoken language visible to the hearing impaired. Previous studies solved asynchronous problems between lip and hand movements by a cuerfootnoteThe people who perform Cued Speech are called the cuer.-dependent piecewise linear model for English and French CS. In this work, we innovatively propose three statistical measure on the lip stream to build an interpretable and generalizable model for predicting hand preceding time (HPT), which achieves cuer-independent by a proper normalization. Particularly, we build the first Mandarin CS corpus comprising annotated videos from five speakers including three normal and two hearing impaired individuals. Consequently, we show that the hand preceding phenomenon exists in Mandarin CS production with significant differences between normal and hearing impaired people. Extensive experiments demonstrate that our model outperforms the baseline and the previous state-of-the-art methods.