# Investigating the dynamics of hand and lips in French Cued Speech using attention mechanisms and CTC-based decoding

arXiv: https://arxiv.org/abs/2306.08290 · HTML: https://mlsanity.com/paper/2306.08290/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 26 references that are in the map.
- First posted on arXiv 14 Jun 2023.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Word Discovery in Visually Grounded, Self-Supervised Speech Models](https://mlsanity.com/paper/2203.15081/) (2022)
- [Re-synchronization using the Hand Preceding Model for Multi-modal Fusion in Automatic Continuous Cued Speech Recognition](https://mlsanity.com/paper/2001.00854/) (2020)
- [Cross-Modal Knowledge Distillation Method for Automatic Cued Speech Recognition](https://mlsanity.com/paper/2106.13686/) (2021)
- [Cross-Modal Mutual Learning for Cued Speech Recognition](https://mlsanity.com/paper/2212.01083/) (2022)
- [Multistream neural architectures for cued-speech recognition using a pre-trained visual feature extractor and constrained CTC decoding](https://mlsanity.com/paper/2204.04965/) (2022)

## Built on by (most cited first, 1 of 1)

- [Cued Speech Generation Leveraging a Pre-trained Audiovisual Text-to-Speech Model](https://mlsanity.com/paper/2501.04799/) (2025)

## Abstract (arXiv)

Hard of hearing or profoundly deaf people make use of cued speech (CS) as a communication tool to understand spoken language. By delivering cues that are relevant to the phonetic information, CS offers a way to enhance lipreading. In literature, there have been several studies on the dynamics between the hand and the lips in the context of human production. This article proposes a way to investigate how a neural network learns this relation for a single speaker while performing a recognition task using attention mechanisms. Further, an analysis of the learnt dynamics is utilized to establish the relationship between the two modalities and extract automatic segments. For the purpose of this study, a new dataset has been recorded for French CS. Along with the release of this dataset, a benchmark will be reported for word-level recognition, a novelty in the automatic recognition of French CS.