# Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model

arXiv: https://arxiv.org/abs/2310.14946 · HTML: https://mlsanity.com/paper/2310.14946/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2023.

## Built on by (most cited first, 4 of 4)

- [mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition](https://mlsanity.com/paper/2502.01547/) (2025)
- [Tailored Design of Audio-Visual Speech Recognition Models using Branchformers](https://mlsanity.com/paper/2407.06606/) (2024)
- [Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extraction](https://mlsanity.com/paper/2506.09792/) (2025)
- [Testing chatbots on the creation of encoders for audio conditioned image generation](https://mlsanity.com/paper/2509.09717/) (2025)

## Abstract (arXiv)

We present a novel approach to multilingual audio-visual speech recognition tasks by introducing a single model on a multilingual dataset. Motivated by a human cognitive system where humans can intuitively distinguish different languages without any conscious effort or guidance, we propose a model that can capture which language is given as an input speech by distinguishing the inherent similarities and differences between languages. To do so, we design a prompt fine-tuning technique into the largely pre-trained audio-visual representation model so that the network can recognize the language class as well as the speech with the corresponding language. Our work contributes to developing robust and efficient multilingual audio-visual speech recognition systems, reducing the need for language-specific models.