# Transcription-Free Fine-Tuning of Speech Separation Models for Noisy and Reverberant Multi-Speaker Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2406.08914 · HTML: https://mlsanity.com/paper/2406.08914/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 33 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Builds on (most cited first, 26 of 26)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [TasNet: time-domain audio separation network for real-time, single-channel speech separation](https://mlsanity.com/paper/1711.00541/) (2017)
- [Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networks](https://mlsanity.com/paper/1703.06284/) (2017)
- [Serialized Output Training for End-to-End Overlapped Speech Recognition](https://mlsanity.com/paper/2003.12687/) (2020)
- [Wavesplit: End-to-End Speech Separation by Speaker Clustering](https://mlsanity.com/paper/2002.08933/) (2020)
- [WHAMR!: Noisy and Reverberant Single-Channel Speech Separation](https://mlsanity.com/paper/1910.10279/) (2019)
- [Single-Channel Multi-Speaker Separation using Deep Clustering](https://mlsanity.com/paper/1607.02173/) (2016)
- [MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognition](https://mlsanity.com/paper/1910.06522/) (2019)
- [Far-Field Automatic Speech Recognition](https://mlsanity.com/paper/2009.09395/) (2020)
- [Multi-turn RNN-T for streaming recognition of multi-party speech](https://mlsanity.com/paper/2112.10200/) (2021)
- [Monaural source separation: From anechoic to reverberant environments](https://mlsanity.com/paper/2111.07578/) (2021)
- [Non-Intrusive Speech Intelligibility Prediction for Hearing-Impaired Users using Intermediate ASR Features and Human Memory Models](https://mlsanity.com/paper/2401.13611/) (2024)
- [Spectral feature mapping with mimic loss for robust speech recognition](https://mlsanity.com/paper/1803.09816/) (2018)
- [Perceive and predict: self-supervised speech representation based loss functions for speech enhancement](https://mlsanity.com/paper/2301.04388/) (2023)
- [PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings](https://mlsanity.com/paper/2403.02288/) (2024)
- [Receptive Field Analysis of Temporal Convolutional Networks for Monaural Speech Dereverberation](https://mlsanity.com/paper/2204.06439/) (2022)
- [Graph-PIT: Generalized permutation invariant training for continuous separation of arbitrary numbers of speakers](https://mlsanity.com/paper/2107.14446/) (2021)
- [On Word Error Rate Definitions and their Efficient Computation for Multi-Speaker Speech Recognition Systems](https://mlsanity.com/paper/2211.16112/) (2022)
- [Meeting Recognition with Continuous Speech Separation and Transcription-Supported Diarization](https://mlsanity.com/paper/2309.16482/) (2023)
- [Exploiting Hidden Representations from a DNN-based Speech Recogniser for Speech Intelligibility Prediction in Hearing-impaired Listeners](https://mlsanity.com/paper/2204.04287/) (2022)
- [On Time Domain Conformer Models for Monaural Speech Separation in Noisy Reverberant Acoustic Environments](https://mlsanity.com/paper/2310.06125/) (2023)
- [On Data Sampling Strategies for Training Neural Network Speech Separation Models](https://mlsanity.com/paper/2304.07142/) (2023)
- [Multi-CMGAN+/+: Leveraging Multi-Objective Speech Quality Metric Prediction for Speech Enhancement](https://mlsanity.com/paper/2312.08979/) (2023)

## Built on by (most cited first, 1 of 1)

- [Whilter: A Whisper-based Data Filter for "In-the-Wild" Speech Corpora Using Utterance-level Multi-Task Classification](https://mlsanity.com/paper/2507.21642/) (2025)

## Abstract (arXiv)

One solution to automatic speech recognition (ASR) of overlapping speakers is to separate speech and then perform ASR on the separated signals. Commonly, the separator produces artefacts which often degrade ASR performance. Addressing this issue typically requires reference transcriptions to jointly train the separation and ASR networks. This is often not viable for training on real-world in-domain audio where reference transcript information is not always available. This paper proposes a transcription-free method for joint training using only audio signals. The proposed method uses embedding differences of pre-trained ASR encoders as a loss with a proposed modification to permutation invariant training (PIT) called guided PIT (GPIT). The method achieves a 6.4% improvement in word error rate (WER) measures over a signal-level loss and also shows enhancement improvements in perceptual measures such as short-time objective intelligibility (STOI).