# AnnoTheia: A Semi-Automatic Annotation Toolkit for Audio-Visual Speech Technologies

arXiv: https://arxiv.org/abs/2402.13152 · HTML: https://mlsanity.com/paper/2402.13152/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 64 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation](https://mlsanity.com/paper/1804.03619/) (2018)
- [End-to-end Audio-visual Speech Recognition with Conformers](https://mlsanity.com/paper/2102.06657/) (2021)
- [Direct speech-to-speech translation with discrete units](https://mlsanity.com/paper/2107.05604/) (2021)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [Direct speech-to-speech translation with a sequence-to-sequence model](https://mlsanity.com/paper/1904.06037/) (2019)
- [DSFD: Dual Shot Face Detector](https://mlsanity.com/paper/1810.10220/) (2018)
- [Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection](https://mlsanity.com/paper/2107.06592/) (2021)
- [The Multilingual TEDx Corpus for Speech Recognition and Translation](https://mlsanity.com/paper/2102.01757/) (2021)
- [Visual Speech Recognition for Multiple Languages in the Wild](https://mlsanity.com/paper/2202.13084/) (2022)
- [End-to-End Neural Speaker Diarization with Permutation-Free Objectives](https://mlsanity.com/paper/1909.05952/) (2019)
- [AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection](https://mlsanity.com/paper/1901.01342/) (2019)
- [End-to-end speaker segmentation for overlap-aware resegmentation](https://mlsanity.com/paper/2104.04045/) (2021)

## Built on by (most cited first, 1 of 1)

- [Evaluation of End-to-End Continuous Spanish Lipreading in Different Data Conditions](https://mlsanity.com/paper/2502.00464/) (2025)

## Abstract (arXiv)

More than 7,000 known languages are spoken around the world. However, due to the lack of annotated resources, only a small fraction of them are currently covered by speech technologies. Albeit self-supervised speech representations, recent massive speech corpora collections, as well as the organization of challenges, have alleviated this inequality, most studies are mainly benchmarked on English. This situation is aggravated when tasks involving both acoustic and visual speech modalities are addressed. In order to promote research on low-resource languages for audio-visual speech technologies, we present AnnoTheia, a semi-automatic annotation toolkit that detects when a person speaks on the scene and the corresponding transcription. In addition, to show the complete process of preparing AnnoTheia for a language of interest, we also describe the adaptation of a pre-trained model for active speaker detection to Spanish, using a database not initially conceived for this type of task. The AnnoTheia toolkit, tutorials, and pre-trained models are available on GitHub.