mlsanityData to 7 Oct 2026

Speech Separation based on Contrastive Learning and Deep Modularization

arXiv 2305.10652 ↗First posted 18 May 2023cs.SD cs.CL eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

The current monaural state of the art tools for speech separation relies on supervised learning. This means that they must deal with permutation problem, they are impacted by the mismatch on the number of speakers used in training and inference. Moreover, their performance heavily relies on the presence of high-quality labelled data. These problems can be effectively addressed by employing a fully unsupervised technique for speech separation. In this paper, we use contrastive learning to establish the representations of frames then use the learned representations in the downstream deep modularization task. Concretely, we demonstrate experimentally that in speech separation, different frames of a speaker can be viewed as augmentations of a given hidden standard frame of that speaker. The frames of a speaker contain enough prosodic information overlap which is key in speech separation. Based on this, we implement a self-supervised learning to learn to minimize the distance between frames belonging to a given speaker. The learned representations are used in a downstream deep modularization task to cluster frames based on speaker identity. Evaluation of the developed technique on WSJ0-2mix and WSJ0-3mix shows that the technique attains SI-SNRi and SDRi of 20.8 and 21.0 respectively in WSJ0-2mix. In WSJ0-3mix, it attains SI-SNRi and SDRi of 20.7 and 20.7 respectively in WSJ0-2mix. Its greatest strength being that as the number of speakers increase, its performance does not degrade significantly.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 59 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

35 of 59 references.

References · 35

  1. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  2. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 400
  3. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 235
  4. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 211
  5. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 189
  6. 2020Attention is All You Need in Speech Separationcited by 184
  7. 2019Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separationcited by 182
  8. 2020Graph Clustering with Graph Neural Networkscited by 163
  9. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 154
  10. 2017TasNet: time-domain audio separation network for real-time, single-channel speech separationcited by 115
  11. 2019MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancementcited by 104
  12. 2020Wavesplit: End-to-End Speech Separation by Speaker Clusteringcited by 74
23 more
  1. 2022TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separationcited by 62
  2. 2019WHAMR!: Noisy and Reverberant Single-Channel Speech Separationcited by 61
  3. 2020Voice Separation with an Unknown Number of Multiple Speakerscited by 46
  4. 2022VoiceFixer: A Unified Framework for High-Fidelity Speech Restorationcited by 37
  5. 2022Investigating self-supervised learning for speech enhancement and separationcited by 35
  6. 2019Divide and Conquer: A Deep CASA Approach to Talker-independent Monaural Speaker Separationcited by 20
  7. 2022RemixIT: Continual self-training of speech enhancement models via bootstrapped remixingcited by 20
  8. 2017Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognitioncited by 19
  9. 2017Complex spectrogram enhancement by convolutional neural network with multi-metrics learningcited by 17
  10. 2019Recursive speech separation for unknown number of speakerscited by 14
  11. 2018Deep Learning Based Phase Reconstruction for Speaker Separation: A Trigonometric Perspectivecited by 13
  12. 2021Many-Speakers Single Channel Speech Separation with Optimal Permutation Trainingcited by 11
  13. 2017Single Channel Audio Source Separation using Convolutional Denoising Autoencoderscited by 8
  14. 2020Towards Listening to 10 People Simultaneously: An Efficient Permutation Invariant Training of Audio Source Separation Using Sinkhorn's Algorithmcited by 8
  15. 2021Teacher-Student MixIT for Unsupervised and Semi-supervised Speech Separationcited by 8
  16. 2016Speech Enhancement In Multiple-Noise Conditions using Deep Neural Networkscited by 7
  17. 2021Unsupervised Speech Enhancement with speech recognition embedding and disentanglement lossescited by 7
  18. 2022MixCycle: Unsupervised Speech Separation via Cyclic Mixture Permutation Invariant Trainingcited by 7
  19. 2021Training Speech Enhancement Systems with Noisy Speech Datasetscited by 6
  20. 2022SkipConvGAN: Monaural Speech Dereverberation using Generative Adversarial Networks via Complex Time-Frequency Maskingcited by 6
  21. 2022Complex-Valued Time-Frequency Self-Attention for Speech Dereverberationcited by 6
  22. 2017Integrated Speech Enhancement Method Based on Weighted Prediction Error and DNN for Dereverberation and Denoisingcited by 2
  23. 2020Separating Varying Numbers of Sources with Auxiliary Autoencoding Losscited by 2

Built on by

Not cited yet.