# Speaker-Targeted Audio-Visual Models for Speech Recognition in Cocktail-Party Environments

arXiv: https://arxiv.org/abs/1906.05962 · HTML: https://mlsanity.com/paper/1906.05962/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2019.

## Built on by (most cited first, 5 of 5)

- [Detector-Free Weakly Supervised Grounding by Separation](https://mlsanity.com/paper/2104.09829/) (2021)
- [Cocktail-Party Audio-Visual Speech Recognition](https://mlsanity.com/paper/2506.02178/) (2025)
- [End-to-end multi-talker audio-visual ASR using an active speaker attention module](https://mlsanity.com/paper/2204.00652/) (2022)
- [Motif-topology improved Spiking Neural Network for the Cocktail Party Effect and McGurk Effect](https://mlsanity.com/paper/2211.07641/) (2022)
- [From Speech to Interaction: Analyzing Multimodal Systems in Cocktail-Party Scenarios](https://mlsanity.com/paper/2608.08510/) (2026)

## Abstract (arXiv)

Speech recognition in cocktail-party environments remains a significant challenge for state-of-the-art speech recognition systems, as it is extremely difficult to extract an acoustic signal of an individual speaker from a background of overlapping speech with similar frequency and temporal characteristics. We propose the use of speaker-targeted acoustic and audio-visual models for this task. We complement the acoustic features in a hybrid DNN-HMM model with information of the target speaker's identity as well as visual features from the mouth region of the target speaker. Experimentation was performed using simulated cocktail-party data generated from the GRID audio-visual corpus by overlapping two speakers's speech on a single acoustic channel. Our audio-only baseline achieved a WER of 26.3%. The audio-visual model improved the WER to 4.4%. Introducing speaker identity information had an even more pronounced effect, improving the WER to 3.6%. Combining both approaches, however, did not significantly improve performance further. Our work demonstrates that speaker-targeted models can significantly improve the speech recognition in cocktail party environments.