# Learning to Enhance or Not: Neural Network-Based Switching of Enhanced and Observed Signals for Overlapping Speech Recognition

arXiv: https://arxiv.org/abs/2201.03881 · HTML: https://mlsanity.com/paper/2201.03881/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jan 2022.

## Built on by (most cited first, 9 of 9)

- [Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation model](https://mlsanity.com/paper/2304.11976/) (2023)
- [Noise robust speech emotion recognition with signal-to-noise ratio adapting speech enhancement](https://mlsanity.com/paper/2309.01164/) (2023)
- [Dual-Branch Knowledge Distillation for Noise-Robust Synthetic Speech Detection](https://mlsanity.com/paper/2310.08869/) (2023)
- [Handling Trade-Offs in Speech Separation with Sparsely-Gated Mixture of Experts](https://mlsanity.com/paper/2211.06493/) (2022)
- [Knowledge Distillation for Neural Transducer-based Target-Speaker ASR: Exploiting Parallel Mixture/Single-Talker Speech Data](https://mlsanity.com/paper/2305.15971/) (2023)
- [ElectrodeNet -- A Deep Learning Based Sound Coding Strategy for Cochlear Implants](https://mlsanity.com/paper/2305.16753/) (2023)
- [LC4SV: A Denoising Framework Learning to Compensate for Unseen Speaker Verification Models](https://mlsanity.com/paper/2311.16604/) (2023)
- [Training-Free Intelligibility-Guided Observation Addition for Noisy ASR](https://mlsanity.com/paper/2602.20967/) (2026)
- [Visual-Informed Speech Enhancement Using Attention-Based Beamforming](https://mlsanity.com/paper/2603.05270/) (2026)

## Abstract (arXiv)

The combination of a deep neural network (DNN) -based speech enhancement (SE) front-end and an automatic speech recognition (ASR) back-end is a widely used approach to implement overlapping speech recognition. However, the SE front-end generates processing artifacts that can degrade the ASR performance. We previously found that such performance degradation can occur even under fully overlapping conditions, depending on the signal-to-interference ratio (SIR) and signal-to-noise ratio (SNR). To mitigate the degradation, we introduced a rule-based method to switch the ASR input between the enhanced and observed signals, which showed promising results. However, the rule's optimality was unclear because it was heuristically designed and based only on SIR and SNR values. In this work, we propose a DNN-based switching method that directly estimates whether ASR will perform better on the enhanced or observed signals. We also introduce soft-switching that computes a weighted sum of the enhanced and observed signals for ASR input, with weights given by the switching model's output posteriors. The proposed learning-based switching showed performance comparable to that of rule-based oracle switching. The soft-switching further improved the ASR performance and achieved a relative character error rate reduction of up to 23 % as compared with the conventional method.