mlsanityData to 14 Oct 2026

Exploring the Integration of Speech Separation and Recognition with Self-Supervised Learning Representation

arXiv 2307.12231 ↗First posted 23 Jul 2023cs.SD cs.CL eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

Neural speech separation has made remarkable progress and its integration with automatic speech recognition (ASR) is an important direction towards realizing multi-speaker ASR. This work provides an insightful investigation of speech separation in reverberant and noisy-reverberant scenarios as an ASR front-end. In detail, we explore multi-channel separation methods, mask-based beamforming and complex spectral mapping, as well as the best features to use in the ASR back-end model. We employ the recent self-supervised learning representation (SSLR) as a feature and improve the recognition performance from the case with filterbank features. To further improve multi-speaker recognition performance, we present a carefully designed training strategy for integrating speech separation and recognition with SSLR. The proposed integration using TF-GridNet-based complex spectral mapping and WavLM-based SSLR achieves a 2.5% word error rate in reverberant WHAMR! test set, significantly outperforming an existing mask-based MVDR beamforming and filterbank integration (28.9%).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 46 references that are in the map. First posted on arXiv 23 Jul 2023.

Builds on

31 of 46 references.

References · 31

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  4. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  5. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 406
  6. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 257
  7. 2017Supervised Speech Separation Based on Deep Learning: An Overviewcited by 216
  8. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 212
  9. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 191
  10. 2020Attention is All You Need in Speech Separationcited by 185
  11. 2019Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separationcited by 182
  12. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
19 more
  1. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 77
  2. 2022TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separationcited by 63
  3. 2019WHAMR!: Noisy and Reverberant Single-Channel Speech Separationcited by 61
  4. 2019MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognitioncited by 44
  5. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  6. 2020Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysiscited by 35
  7. 2020Streaming Multi-speaker ASR with RNN-Tcited by 23
  8. 2017Multichannel End-to-end Speech Recognitioncited by 17
  9. 2022End-to-End Integration of Speech Recognition, Speech Enhancement, and Self-Supervised Learning Representationcited by 17
  10. 2020Multi-talker ASR for an unknown number of sources: Joint training of source counting, separation and ASRcited by 15
  11. 2020End-to-End Far-Field Speech Recognition with Unified Dereverberation and Beamformingcited by 9
  12. 2019Frequency Domain Multi-channel Acoustic Modeling for Distant Speech Recognitioncited by 8
  13. 2019End-to-end training of time domain audio separation and recognitioncited by 8
  14. 2022ESPnet-SE++: Speech Enhancement for Robust Speech Recognition, Translation, and Understandingcited by 8
  15. 2020Convolutive Transfer Function Invariant SDR training criteria for Multi-Channel Reverberant Speech Separationcited by 6
  16. 2021Closing the Gap Between Time-Domain Multi-Channel Speech Enhancement on Real and Simulation Conditionscited by 6
  17. 2022End-to-End Integration of Speech Recognition, Dereverberation, Beamforming, and Self-Supervised Learning Representationcited by 6
  18. 2022Towards Low-distortion Multi-channel Speech Enhancement: The ESPNet-SE Submission to The L3DAS22 Challengecited by 4
  19. 2021Time-Domain Speech Extraction with Spatial Information and Multi Speaker Conditioning Mechanismcited by 2

Built on by

Not cited yet.