mlsanityData to 7 Oct 2026

AnnoTheia: A Semi-Automatic Annotation Toolkit for Audio-Visual Speech Technologies

arXiv 2402.13152 ↗First posted 20 Feb 2024cs.CV cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

More than 7,000 known languages are spoken around the world. However, due to the lack of annotated resources, only a small fraction of them are currently covered by speech technologies. Albeit self-supervised speech representations, recent massive speech corpora collections, as well as the organization of challenges, have alleviated this inequality, most studies are mainly benchmarked on English. This situation is aggravated when tasks involving both acoustic and visual speech modalities are addressed. In order to promote research on low-resource languages for audio-visual speech technologies, we present AnnoTheia, a semi-automatic annotation toolkit that detects when a person speaks on the scene and the corresponding transcription. In addition, to show the complete process of preparing AnnoTheia for a language of interest, we also describe the adaptation of a pre-trained model for active speaker detection to Spanish, using a database not initially conceived for this type of task. The AnnoTheia toolkit, tutorials, and pre-trained models are available on GitHub.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 49 of its 64 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

49 of 64 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2017Squeeze-and-Excitation Networkscited by 4,763
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  8. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  9. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  10. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  11. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
  12. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
37 more
  1. 2018Deep Audio-Visual Speech Recognitioncited by 318
  2. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  3. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  4. 2020Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translationcited by 254
  5. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  6. 2017SEGAN: Speech Enhancement Generative Adversarial Networkcited by 165
  7. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 160
  8. 2021Direct speech-to-speech translation with discrete unitscited by 148
  9. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  10. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  11. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  12. 2018DSFD: Dual Shot Face Detectorcited by 115
  13. 2021Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detectioncited by 110
  14. 2021The Multilingual TEDx Corpus for Speech Recognition and Translationcited by 101
  15. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 94
  16. 2019End-to-End Neural Speaker Diarization with Permutation-Free Objectivescited by 86
  17. 2019AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detectioncited by 81
  18. 2021End-to-end speaker segmentation for overlap-aware resegmentationcited by 76
  19. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 70
  20. 2021Sub-word Level Lip Reading With Visual Attentioncited by 60
  21. 2018LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wildcited by 51
  22. 2023MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translationcited by 39
  23. 2023A Light Weight Model for Active Speaker Detectioncited by 39
  24. 2022Investigating self-supervised learning for speech enhancement and separationcited by 35
  25. 2022Cold Diffusion for Speech Enhancementcited by 31
  26. 2023Improving Massively Multilingual ASR With Auxiliary CTC Objectivescited by 31
  27. 2022Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detectioncited by 29
  28. 2019A Cascade Sequence-to-Sequence Model for Chinese Mandarin Lip Readingcited by 25
  29. 2021Face, Body, Voice: Video Person-Clustering with Multiple Modalitiescited by 24
  30. 2023Audio-Visual Efficient Conformer for Robust Speech Recognitioncited by 22
  31. 2018Network Decoupling: From Regular to Depthwise Separable Convolutionscited by 17
  32. 2017Towards Estimating the Upper Bound of Visual-Speech Recognition: The Visual Lip-Reading Feasibility Databasecited by 13
  33. 2020Silent Speech Interfaces for Speech Restoration: A Reviewcited by 13
  34. 2023LIP-RTVE: An Audiovisual Database for Continuous Spanish in the Wildcited by 7
  35. 2023AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translationcited by 6
  36. 2021LRWR: Large-Scale Benchmark for Lip Reading in Russian languagecited by 5
  37. 2023A Multi-Purpose Audio-Visual Corpus for Multi-Modal Persian Speech Recognition: the Arman-AV Datasetcited by 4

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Evaluation of End-to-End Continuous Spanish Lipreading in Different Data Conditionscited by 0