# Convoifilter: A case study of doing cocktail party speech recognition

arXiv: https://arxiv.org/abs/2308.11380 · HTML: https://mlsanity.com/paper/2308.11380/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 22 Aug 2023.

## Builds on (most cited first, 18 of 18)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019)
- [TasNet: time-domain audio separation network for real-time, single-channel speech separation](https://mlsanity.com/paper/1711.00541/) (2017)
- [The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines](https://mlsanity.com/paper/1803.10609/) (2018)
- [Continuous speech separation: dataset and analysis](https://mlsanity.com/paper/2001.11482/) (2020)
- [VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking](https://mlsanity.com/paper/1810.04826/) (2018)
- [Building and Evaluation of a Real Room Impulse Response Dataset](https://mlsanity.com/paper/1811.06795/) (2018)
- [VoiceFilter-Lite: Streaming Targeted Voice Separation for On-Device Speech Recognition](https://mlsanity.com/paper/2009.04323/) (2020)
- [Far-Field Automatic Speech Recognition](https://mlsanity.com/paper/2009.09395/) (2020)
- [Super-Human Performance in Online Low-latency Recognition of Conversational Speech](https://mlsanity.com/paper/2010.03449/) (2020)
- [Deep Extractor Network for Target Speaker Recovery From Single Channel Speech Mixtures](https://mlsanity.com/paper/1807.08974/) (2018)
- [A Comparison of Metric Learning Loss Functions for End-To-End Speaker Verification](https://mlsanity.com/paper/2003.14021/) (2020)
- [An Investigation into the Effectiveness of Enhancement in ASR Training and Test for CHiME-5 Dinner Party Transcription](https://mlsanity.com/paper/1909.12208/) (2019)

## Built on by (most cited first, 2 of 2)

- [MSA-ASR: Efficient Multilingual Speaker Attribution with frozen ASR Models](https://mlsanity.com/paper/2411.18152/) (2024)
- [Mask2Flow-TSE: Two-Stage Target Speaker Extraction with Masking and Flow Matching](https://mlsanity.com/paper/2603.12837/) (2026)

## Abstract (arXiv)

This paper presents an end-to-end model designed to improve automatic speech recognition (ASR) for a particular speaker in a crowded, noisy environment. The model utilizes a single-channel speech enhancement module that isolates the speaker's voice from background noise (ConVoiFilter) and an ASR module. The model can decrease ASR's word error rate (WER) from 80% to 26.4% through this approach. Typically, these two components are adjusted independently due to variations in data requirements. However, speech enhancement can create anomalies that decrease ASR efficiency. By implementing a joint fine-tuning strategy, the model can reduce the WER from 26.4% in separate tuning to 14.5% in joint tuning. We openly share our pre-trained model to foster further research hf.co/nguyenvulebinh/voice-filter.