# A Multimodal German Dataset for Automatic Lip Reading Systems and Transfer Learning

arXiv: https://arxiv.org/abs/2202.13403 · HTML: https://mlsanity.com/paper/2202.13403/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 27 Feb 2022.

## Builds on (most cited first, 12 of 12)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [cuDNN: Efficient Primitives for Deep Learning](https://mlsanity.com/paper/1410.0759/) (2014)
- [Lipreading using Temporal Convolutional Networks](https://mlsanity.com/paper/2001.08702/) (2020)
- [Combining Residual Networks with LSTMs for Lipreading](https://mlsanity.com/paper/1703.04105/) (2017)
- [LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild](https://mlsanity.com/paper/1810.06990/) (2018)
- [PyTorchVideo: A Deep Learning Library for Video Understanding](https://mlsanity.com/paper/2111.09887/) (2021)
- [Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for Lipreading](https://mlsanity.com/paper/1905.02540/) (2019)
- [Privacy issues on biometric systems](https://mlsanity.com/paper/2202.11415/) (2022)
- [The speaker-independent lipreading play-off; a survey of lipreading machines](https://mlsanity.com/paper/1810.10597/) (2018)
- [Visual speech recognition: aligning terminologies for better understanding](https://mlsanity.com/paper/1710.01292/) (2017)

## Built on by (most cited first, 8 of 8)

- [A Multi-Purpose Audio-Visual Corpus for Multi-Modal Persian Speech Recognition: the Arman-AV Dataset](https://mlsanity.com/paper/2301.10180/) (2023)
- [OLKAVS: An Open Large-Scale Korean Audio-Visual Speech Dataset](https://mlsanity.com/paper/2301.06375/) (2023)
- [Word-level Persian Lipreading Dataset](https://mlsanity.com/paper/2304.04068/) (2023)
- [End-to-End Lip Reading in Romanian with Cross-Lingual Domain Adaptation and Lateral Inhibition](https://mlsanity.com/paper/2310.04858/) (2023)
- [Transfer Learning from Visual Speech Recognition to Mouthing Recognition in German Sign Language](https://mlsanity.com/paper/2505.13784/) (2025)
- [Automated Speaker Independent Visual Speech Recognition: A Comprehensive Survey](https://mlsanity.com/paper/2306.08314/) (2023)
- [LRW-Persian: Lip-reading in the Wild Dataset for Persian Language](https://mlsanity.com/paper/2510.22716/) (2025)
- [VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness](https://mlsanity.com/paper/2607.08112/) (2026)

## Abstract (arXiv)

Large datasets as required for deep learning of lip reading do not exist in many languages. In this paper we present the dataset GLips (German Lips) consisting of 250,000 publicly available videos of the faces of speakers of the Hessian Parliament, which was processed for word-level lip reading using an automatic pipeline. The format is similar to that of the English language LRW (Lip Reading in the Wild) dataset, with each video encoding one word of interest in a context of 1.16 seconds duration, which yields compatibility for studying transfer learning between both datasets. By training a deep neural network, we investigate whether lip reading has language-independent features, so that datasets of different languages can be used to improve lip reading models. We demonstrate learning from scratch and show that transfer learning from LRW to GLips and vice versa improves learning speed and performance, in particular for the validation set.