# VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognition arXiv: https://arxiv.org/abs/2209.04974 ยท HTML: https://mlsanity.com/paper/2209.04974/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 26 of its 36 references that are in the map. - First posted on arXiv 12 Sep 2022. ## Builds on (most cited first, 26 of 26) - [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020) - [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021) - [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020) - [Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset](https://mlsanity.com/paper/2010.11395/) (2020) - [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021) - [Serialized Output Training for End-to-End Overlapped Speech Recognition](https://mlsanity.com/paper/2003.12687/) (2020) - [Continuous speech separation: dataset and analysis](https://mlsanity.com/paper/2001.11482/) (2020) - [M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge](https://mlsanity.com/paper/2110.07393/) (2021) - [MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognition](https://mlsanity.com/paper/1910.06522/) (2019) - [Streaming Multi-Talker ASR with Token-Level Serialized Output Training](https://mlsanity.com/paper/2202.00842/) (2022) - [Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysis](https://mlsanity.com/paper/2011.02014/) (2020) - [Recognizing Multi-talker Speech with Permutation Invariant Training](https://mlsanity.com/paper/1704.01985/) (2017) - [Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphone](https://mlsanity.com/paper/2103.16776/) (2021) - [End-to-end Microphone Permutation and Number Invariant Multi-channel Speech Separation](https://mlsanity.com/paper/1910.14104/) (2019) - [A Purely End-to-end System for Multi-speaker Speech Recognition](https://mlsanity.com/paper/1805.05826/) (2018) - [Streaming end-to-end multi-talker speech recognition](https://mlsanity.com/paper/2011.13148/) (2020) - [End-to-End Monaural Multi-speaker ASR System without Pretraining](https://mlsanity.com/paper/1811.02062/) (2018) - [Recognizing Overlapped Speech in Meetings: A Multichannel Separation Approach Using Neural Networks](https://mlsanity.com/paper/1810.03655/) (2018) - [Streaming Multi-speaker ASR with RNN-T](https://mlsanity.com/paper/2011.11671/) (2020) - [Simultaneous Speech Recognition and Speaker Diarization for Monaural Dialogue Recordings with Target-Speaker Acoustic Models](https://mlsanity.com/paper/1909.08103/) (2019) - [End-to-End Dereverberation, Beamforming, and Speech Recognition with Improved Numerical Stability and Advanced Frontend](https://mlsanity.com/paper/2102.11525/) (2021) - [Guided Source Separation Meets a Strong ASR Backend: Hitachi/Paderborn University Joint Investigation for Dinner Party ASR](https://mlsanity.com/paper/1905.12230/) (2019) - [VarArray: Array-Geometry-Agnostic Continuous Speech Separation](https://mlsanity.com/paper/2110.05745/) (2021) - [Leveraging Real Conversational Data for Multi-Channel Continuous Speech Separation](https://mlsanity.com/paper/2204.03232/) (2022) - [The RoyalFlush System of Speech Recognition for M2MeT Challenge](https://mlsanity.com/paper/2202.01614/) (2022) - [The Volcspeech system for the ICASSP 2022 multi-channel multi-party meeting transcription challenge](https://mlsanity.com/paper/2202.04261/) (2022) ## Built on by (most cited first, 7 of 7) - [The CHiME-7 DASR Challenge: Distant Meeting Transcription with Multiple Devices in Diverse Scenarios](https://mlsanity.com/paper/2306.13734/) (2023) - [One model to rule them all ? Towards End-to-End Joint Speaker Diarization and Speech Recognition](https://mlsanity.com/paper/2310.01688/) (2023) - [Simulating realistic speech overlaps improves multi-talker ASR](https://mlsanity.com/paper/2210.15715/) (2022) - [Token-Level Serialized Output Training for Joint Streaming ASR and ST Leveraging Textual Alignments](https://mlsanity.com/paper/2307.03354/) (2023) - [Neural Blind Source Separation and Diarization for Distant Speech Recognition](https://mlsanity.com/paper/2406.08396/) (2024) - [Joint Beamforming and Speaker-Attributed ASR for Real Distant-Microphone Meeting Transcription](https://mlsanity.com/paper/2410.21849/) (2024) - [Improving Practical Aspects of End-to-End Multi-Talker Speech Recognition for Online and Offline Scenarios](https://mlsanity.com/paper/2506.14204/) (2025) ## Abstract (arXiv) This paper presents a novel streaming automatic speech recognition (ASR) framework for multi-talker overlapping speech captured by a distant microphone array with an arbitrary geometry. Our framework, named t-SOT-VA, capitalizes on independently developed two recent technologies; array-geometry-agnostic continuous speech separation, or VarArray, and streaming multi-talker ASR based on token-level serialized output training (t-SOT). To combine the best of both technologies, we newly design a t-SOT-based ASR model that generates a serialized multi-talker transcription based on two separated speech signals from VarArray. We also propose a pre-training scheme for such an ASR model where we simulate VarArray's output signals based on monaural single-talker ASR training data. Conversation transcription experiments using the AMI meeting corpus show that the system based on the proposed framework significantly outperforms conventional ones. Our system achieves the state-of-the-art word error rates of 13.7% and 15.5% for the AMI development and evaluation sets, respectively, in the multiple-distant-microphone setting while retaining the streaming inference capability.