# End-to-end Multichannel Speaker-Attributed ASR: Speaker Guided Decoder and Input Feature Analysis

arXiv: https://arxiv.org/abs/2310.10106 · HTML: https://mlsanity.com/paper/2310.10106/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 27 references that are in the map.
- First posted on arXiv 16 Oct 2023.

## Builds on (most cited first, 22 of 22)

- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Serialized Output Training for End-to-End Overlapped Speech Recognition](https://mlsanity.com/paper/2003.12687/) (2020)
- [gpuRIR: A Python Library for Room Impulse Response Simulation with GPU Acceleration](https://mlsanity.com/paper/1810.11359/) (2018)
- [MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognition](https://mlsanity.com/paper/1910.06522/) (2019)
- [End-to-End Multi-speaker Speech Recognition with Transformer](https://mlsanity.com/paper/2002.03921/) (2020)
- [Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakers](https://mlsanity.com/paper/2006.10930/) (2020)
- [End-to-End Speaker-Attributed ASR with Transformer](https://mlsanity.com/paper/2104.02128/) (2021)
- [Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphone](https://mlsanity.com/paper/2103.16776/) (2021)
- [VoxSRC 2020: The Second VoxCeleb Speaker Recognition Challenge](https://mlsanity.com/paper/2012.06867/) (2020)
- [Streaming Multi-speaker ASR with RNN-T](https://mlsanity.com/paper/2011.11671/) (2020)
- [Multi-modal Multi-channel Target Speech Separation](https://mlsanity.com/paper/2003.07032/) (2020)
- [ADL-MVDR: All deep learning MVDR beamformer for target speech separation](https://mlsanity.com/paper/2008.06994/) (2020)
- [End-to-End Multi-Channel Transformer for Speech Recognition](https://mlsanity.com/paper/2102.03951/) (2021)
- [VoxSRC 2019: The first VoxCeleb Speaker Recognition Challenge](https://mlsanity.com/paper/1912.02522/) (2019)
- [Streaming Multi-talker Speech Recognition with Joint Speaker Identification](https://mlsanity.com/paper/2104.02109/) (2021)
- [Multi-Speaker ASR Combining Non-Autoregressive Conformer CTC and Conditional Speaker Chain](https://mlsanity.com/paper/2106.08595/) (2021)
- [Multi-Channel Multi-Speaker ASR Using 3D Spatial Feature](https://mlsanity.com/paper/2111.11023/) (2021)
- [MFCCA:Multi-Frame Cross-Channel attention for multi-speaker ASR in Multi-party meeting scenario](https://mlsanity.com/paper/2210.05265/) (2022)
- [A Comparative Study on Multichannel Speaker-Attributed Automatic Speech Recognition in Multi-party Meetings](https://mlsanity.com/paper/2211.00511/) (2022)
- [End-to-End Multi-speaker ASR with Independent Vector Analysis](https://mlsanity.com/paper/2204.00218/) (2022)

## Built on by (most cited first, 3 of 3)

- [Improving Speaker Assignment in Speaker-Attributed ASR for Real Meeting Applications](https://mlsanity.com/paper/2403.06570/) (2024)
- [Joint Beamforming and Speaker-Attributed ASR for Real Distant-Microphone Meeting Transcription](https://mlsanity.com/paper/2410.21849/) (2024)
- [Exposing the Cost of Deep Learning Audio Development](https://mlsanity.com/paper/2610.01619/) (2026)

## Abstract (arXiv)

We present an end-to-end multichannel speaker-attributed automatic speech recognition (MC-SA-ASR) system that combines a Conformer-based encoder with multi-frame crosschannel attention and a speaker-attributed Transformer-based decoder. To the best of our knowledge, this is the first model that efficiently integrates ASR and speaker identification modules in a multichannel setting. On simulated mixtures of LibriSpeech data, our system reduces the word error rate (WER) by up to 12% and 16% relative compared to previously proposed single-channel and multichannel approaches, respectively. Furthermore, we investigate the impact of different input features, including multichannel magnitude and phase information, on the ASR performance. Finally, our experiments on the AMI corpus confirm the effectiveness of our system for real-world multichannel meeting transcription.