Transcription and translation of videos using fine-tuned XLSR Wav2Vec2 on custom dataset and mBART
Not on the map. Why?
Abstract (arXiv)
This research addresses the challenge of training an ASR model for personalized voices with minimal data. Utilizing just 14 minutes of custom audio from a YouTube video, we employ Retrieval-Based Voice Conversion (RVC) to create a custom Common Voice 16.0 corpus. Subsequently, a Cross-lingual Self-supervised Representations (XLSR) Wav2Vec2 model is fine-tuned on this dataset. The developed web-based GUI efficiently transcribes and translates input Hindi videos. By integrating XLSR Wav2Vec2 and mBART, the system aligns the translated text with the video timeline, delivering an accessible solution for multilingual video content transcription and translation for personalized voice.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 8 of its 12 references that are in the map. First posted on arXiv 1 Mar 2024.
Builds on
8 of 12 references.
References · 8
- 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
- 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
- 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
- 2018A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decaycited by 223
- 2021End-to-end speaker segmentation for overlap-aware resegmentationcited by 76
- 2021Exploring Wav2vec 2.0 fine-tuning for improved speech emotion recognitioncited by 47
- 2020Applying Wav2vec2.0 to Speech Recognition in Various Low-resource Languagescited by 30
- 2022Applying wav2vec2 for Speech Recognition on Bengali Common Voices Datasetcited by 4
Built on by
Not cited yet.