# When Audio Separation Hurts Zero-Shot ASR: Evaluating SAM-Audio with Whisper on Bengali and English Speech

arXiv: https://arxiv.org/abs/2603.04710 · HTML: https://mlsanity.com/paper/2603.04710/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 16 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Builds on (most cited first, 9 of 9)

- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement](https://mlsanity.com/paper/2104.03538/) (2021)
- [A scalable noisy speech dataset and online subjective test framework](https://mlsanity.com/paper/1909.08050/) (2019)
- [SAM Audio: Segment Anything in Audio](https://mlsanity.com/paper/2512.18099/) (2025)
- [Rethinking Processing Distortions: Disentangling the Impact of Speech Enhancement Errors on Speech Recognition Performance](https://mlsanity.com/paper/2404.14860/) (2024)
- [Should We Always Separate?: Switching Between Enhanced and Observed Signals for Overlapping Speech Recognition](https://mlsanity.com/paper/2106.00949/) (2021)
- [A Neural Acoustic Echo Canceller Optimized Using An Automatic Speech Recognizer And Large Scale Synthetic Data](https://mlsanity.com/paper/2106.00856/) (2021)
- [When De-noising Hurts: A Systematic Study of Speech Enhancement Effects on Modern Medical ASR Systems](https://mlsanity.com/paper/2512.17562/) (2025)

## Abstract (arXiv)

Recent advances in automatic speech recognition (ASR) and speech enhancement have strengthened the common belief that cleaner audio should lead to more accurate transcription. In this work, we examine whether this assumption holds for modern zero-shot ASR systems. We conduct a structured empirical study of SAM-Audio as a preprocessing step for zero-shot transcription with OpenAI Whisper. Five Whisper variants are evaluated on noisy Bengali and English speech datasets. On the English dataset, SAM-Audio increases the average PSNR from 32.28 dB to 35.99 dB and achieves higher PSNR for 71.84% of the utterances. However, WER and CER increase in every evaluated model-dataset configuration. On the Bengali dataset, Whisper large-v3 WER increases from 65.83% to 77.35%, while CER increases from 24.13% to 34.74%. On the English dataset, Whisper base WER increases from 10.53% to 21.66%, while CER increases from 4.48% to 12.50%. Utterance-level analysis further shows that the degradation affects a substantial portion of the evaluated samples, although its severity varies across Whisper variants. These findings demonstrate that improved signal-level quality does not necessarily lead to better zero-shot ASR performance and that denoising can reduce recognition accuracy.