mlsanityData to 7 Oct 2026

Recognizing Multi-talker Speech with Permutation Invariant Training

arXiv 1704.01985 ↗First posted 22 Mar 2017cs.SD cs.LG eess.AScited by 33

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a novel technique for direct recognition of multiple speech streams given the single channel of mixed speech, without first separating them. Our technique is based on permutation invariant training (PIT) for automatic speech recognition (ASR). In PIT-ASR, we compute the average cross entropy (CE) over all frames in the whole utterance for each possible output-target assignment, pick the one with the minimum CE, and optimize for that assignment. PIT-ASR forces all the frames of the same speaker to be aligned with the same output layer. This strategy elegantly solves the label permutation problem and speaker tracing problem in one shot. Our experiments on artificially mixed AMI data showed that the proposed approach is very promising.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Mar 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 8; 2023: 5; 2024: 4; 2025: 5; 2026: 4.

33 papers.

Sort:

By area:

  1. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  2. 2021A Review of Speaker Diarization: Recent Advances with Deep Learningcited by 122
  3. 2022Streaming Multi-Talker ASR with Token-Level Serialized Output Trainingcited by 37
  4. 2021End-to-End Speaker-Attributed ASR with Transformercited by 35
  5. 2021Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphonecited by 33
  6. 2022Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddingscited by 14
  7. 2024Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systemscited by 12
  8. 2023Adapting Multi-Lingual ASR Models for Handling Multiple Talkerscited by 11
  9. 2021A Comparative Study of Modular and Joint Approaches for Speaker-Attributed ASR on Monaural Long-Form Audiocited by 10
  10. 2021Multi-turn RNN-T for streaming recognition of multi-party speechcited by 10
  11. 2022Simulating realistic speech overlaps improves multi-talker ASRcited by 10
  12. 2024Advancing Multi-talker ASR Performance with Large Language Modelscited by 10
21 more
  1. 2022VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognitioncited by 7
  2. 2022A Comparative Study on Speaker-attributed Automatic Speech Recognition in Multi-party Meetingscited by 5
  3. 2023BA-SOT: Boundary-Aware Serialized Output Training for Multi-Talker ASRcited by 5
  4. 2021Unified Autoregressive Modeling for Joint End-to-End Multi-Talker Overlapped Speech Recognition and Speaker Attribute Estimationcited by 4
  5. 2023End-to-End Joint Target and Non-Target Speakers ASRcited by 4
  6. 2024Disentangling Speakers in Multi-Talker Speech Recognition with Speaker-Aware CTCcited by 4
  7. 2025Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audiocited by 4
  8. 2022The RoyalFlush System of Speech Recognition for M2MeT Challengecited by 3
  9. 2022Separator-Transducer-Segmenter: Streaming Recognition and Segmentation of Multi-party Speechcited by 3
  10. 2024LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarizationcited by 3
  11. 2025Speaker-Distinguishable CTC: Learning Speaker Distinction Using CTC for Multi-Talker Speech Recognitioncited by 3
  12. 2023Speaker Mask Transformer for Multi-talker Overlapped Speech Recognitioncited by 2
  13. 2026SE-DiCoW: Self-Enrolled Diarization-Conditioned Whispercited by 2
  14. 2022High-resolution embedding extractor for speaker diarisationcited by 1
  15. 2023A Glance is Enough: Extract Target Sentence By Looking at A keywordcited by 1
  16. 2025GLAD: Global-Local Aware Dynamic Mixture-of-Experts for Multi-Talker ASRcited by 1
  17. 2025SC-SOT: Conditioning the Decoder on Diarized Speaker Information for End-to-End Overlapped Speech Recognitioncited by 0
  18. 2025Improving Practical Aspects of End-to-End Multi-Talker Speech Recognition for Online and Offline Scenarioscited by 0
  19. 2026Modeling Overlapped Speech with Shufflescited by 0
  20. 2026H-SAGE: Holistic Speaker-Aware Guided Experts for MoE-based Multi-Talker ASRcited by 0
  21. 2026Who Said What, and Will It Be Remembered? Evaluating Persistent Speaker Attribution Across Meetingscited by 0