mlsanityData to 7 Oct 2026

Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation

arXiv 1607.00325 ↗First posted 1 Jul 2016cs.CL cs.LG cs.SDcited by 189

Not on the map. Why?

Abstract (arXiv)

We propose a novel deep learning model, which supports permutation invariant training (PIT), for speaker independent multi-talker speech separation, commonly known as the cocktail-party problem. Different from most of the prior arts that treat speech separation as a multi-class regression problem and the deep clustering technique that considers it a segmentation (or clustering) problem, our model optimizes for the separation regression error, ignoring the order of mixing sources. This strategy cleverly solves the long-lasting label permutation problem that has prevented progress on deep learning based techniques for speech separation. Experiments on the equal-energy mixing setup of a Danish corpus confirms the effectiveness of PIT. We believe improvements built upon PIT can eventually solve the cocktail-party problem and enable real-world adoption of, e.g., automatic meeting transcription and multi-party human-computer interaction, where overlapping speech is common.

Key facts. Cited by 189 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Jul 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 40; 2022: 43; 2023: 38; 2024: 30; 2025: 23; 2026: 15.

189 papers.

Sort:

By area:

  1. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  2. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  3. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 76
  4. 2022Audio Self-supervised Learning: A Surveycited by 65
  5. 2023Generative Pre-training for Speech with Flow Matchingcited by 55
  6. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  7. 2021A Survey of Sound Source Localization with Deep Learning Methodscited by 45
  8. 2023Separate Anything You Describecited by 43
  9. 2022An efficient encoder-decoder architecture with top-down attention for speech separationcited by 38
  10. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 38
  11. 2022Streaming Multi-Talker ASR with Token-Level Serialized Output Trainingcited by 37
  12. 2021Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphonecited by 33
38 more
  1. 2021Visual Scene Graphs for Audio Source Separationcited by 29
  2. 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
  3. 2021Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separationcited by 27
  4. 2022Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesiscited by 24
  5. 2022CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videoscited by 23
  6. 2022Real-Time Target Sound Extractioncited by 22
  7. 2022RemixIT: Continual self-training of speech enhancement models via bootstrapped remixingcited by 20
  8. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  9. 2024Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructionscited by 17
  10. 2021Compute and memory efficient universal sound source separationcited by 15
  11. 2021Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractorscited by 14
  12. 2023Separate And Diffuse: Using a Pretrained Diffusion Model for Improving Source Separationcited by 14
  13. 2023Extending Whisper with prompt tuning to target-speaker ASRcited by 14
  14. 2022Heterogeneous Target Speech Separationcited by 13
  15. 2023A Sidecar Separator Can Convert a Single-Talker Speech Recognition System to a Multi-Talker Onecited by 13
  16. 2021Streaming Multi-talker Speech Recognition with Joint Speaker Identificationcited by 12
  17. 2023Resilient Multiple Choice Learning: A learned scoring scheme with application to audio scene analysiscited by 12
  18. 2024Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systemscited by 12
  19. 2021Many-Speakers Single Channel Speech Separation with Optimal Permutation Trainingcited by 11
  20. 2021Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problemcited by 11
  21. 2023Adapting Multi-Lingual ASR Models for Handling Multiple Talkerscited by 11
  22. 2023IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech Separationcited by 11
  23. 2024Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition Systemcited by 11
  24. 2021Sandglasset: A Light Multi-Granularity Self-attentive Network For Time-Domain Speech Separationcited by 10
  25. 2021Multi-turn RNN-T for streaming recognition of multi-party speechcited by 10
  26. 2023ReZero: Region-customizable Sound Extractioncited by 10
  27. 2022An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuitscited by 9
  28. 2024Cross-Speaker Encoding Network for Multi-Talker Speech Recognitioncited by 9
  29. 2024Noise-robust Speech Separation with Fast Generative Correctioncited by 9
  30. 2021Effective Low-Cost Time-Domain Audio Separation Using Globally Attentive Locally Recurrent Networkscited by 8
  31. 2022Adversarial Permutation Invariant Training for Universal Sound Separationcited by 8
  32. 2023TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognitioncited by 8
  33. 2024OmniSep: Unified Omni-Modality Sound Separation with Query-Mixupcited by 8
  34. 2021PILOT: Introducing Transformers for Probabilistic Sound Event Localizationcited by 7
  35. 2022MixCycle: Unsupervised Speech Separation via Cyclic Mixture Permutation Invariant Trainingcited by 7
  36. 2022Target Speaker Voice Activity Detection with Transformers and Its Integration with End-to-End Neural Diarizationcited by 7
  37. 2023Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separatorcited by 7
  38. 2024Serialized Speech Information Guidance with Overlapped Encoding Separation for Multi-Speaker Automatic Speech Recognitioncited by 7

50 most cited of 189 All 189 (JSON)