# Transfer Learning from Visual Speech Recognition to Mouthing Recognition in German Sign Language

arXiv: https://arxiv.org/abs/2505.13784 · HTML: https://mlsanity.com/paper/2505.13784/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 30 references that are in the map.
- First posted on arXiv 20 May 2025.

## Builds on (most cited first, 14 of 14)

- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues](https://mlsanity.com/paper/2007.12131/) (2020)
- [Skeleton Aware Multi-modal Sign Language Recognition](https://mlsanity.com/paper/2103.08833/) (2021)
- [Visual Speech Recognition for Multiple Languages in the Wild](https://mlsanity.com/paper/2202.13084/) (2022)
- [LipNet: End-to-End Sentence-level Lipreading](https://mlsanity.com/paper/1611.01599/) (2016)
- [Quantitative Survey of the State of the Art in Sign Language Recognition](https://mlsanity.com/paper/2008.09918/) (2020)
- [PyTorchVideo: A Deep Learning Library for Video Understanding](https://mlsanity.com/paper/2111.09887/) (2021)
- [Deep Learning for Visual Speech Analysis: A Survey](https://mlsanity.com/paper/2205.10839/) (2022)
- [The Role of Pre-training Data in Transfer Learning](https://mlsanity.com/paper/2302.13602/) (2023)
- [A Multimodal German Dataset for Automatic Lip Reading Systems and Transfer Learning](https://mlsanity.com/paper/2202.13403/) (2022)
- [Development and evaluation of a deep learning algorithm for German word recognition from lip movements](https://mlsanity.com/paper/2504.15792/) (2025)

## Built on by (most cited first, 1 of 1)

- [The Importance of Facial Features in Vision-based Sign Language Recognition: Eyes, Mouth or Full Face?](https://mlsanity.com/paper/2507.20884/) (2025)

## Abstract (arXiv)

Sign Language Recognition (SLR) systems primarily focus on manual gestures, but non-manual features such as mouth movements, specifically mouthing, provide valuable linguistic information. This work directly classifies mouthing instances to their corresponding words in the spoken language while exploring the potential of transfer learning from Visual Speech Recognition (VSR) to mouthing recognition in German Sign Language. We leverage three VSR datasets: one in English, one in German with unrelated words and one in German containing the same target words as the mouthing dataset, to investigate the impact of task similarity in this setting. Our results demonstrate that multi-task learning improves both mouthing recognition and VSR accuracy as well as model robustness, suggesting that mouthing recognition should be treated as a distinct but related task to VSR. This research contributes to the field of SLR by proposing knowledge transfer from VSR to SLR datasets with limited mouthing annotations.