# Joint Beamforming and Speaker-Attributed ASR for Real Distant-Microphone Meeting Transcription

arXiv: https://arxiv.org/abs/2410.21849 · HTML: https://mlsanity.com/paper/2410.21849/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 41 references that are in the map.
- First posted on arXiv 29 Oct 2024.

## Builds on (most cited first, 27 of 27)

- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [Serialized Output Training for End-to-End Overlapped Speech Recognition](https://mlsanity.com/paper/2003.12687/) (2020)
- [TorchAudio: Building Blocks for Audio and Speech Processing](https://mlsanity.com/paper/2110.15018/) (2021)
- [M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge](https://mlsanity.com/paper/2110.07393/) (2021)
- [gpuRIR: A Python Library for Room Impulse Response Simulation with GPU Acceleration](https://mlsanity.com/paper/1810.11359/) (2018)
- [Asteroid: the PyTorch-based audio source separation toolkit for researchers](https://mlsanity.com/paper/2005.04132/) (2020)
- [MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognition](https://mlsanity.com/paper/1910.06522/) (2019)
- [Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakers](https://mlsanity.com/paper/2006.10930/) (2020)
- [End-to-End Speaker-Attributed ASR with Transformer](https://mlsanity.com/paper/2104.02128/) (2021)
- [End-to-end Microphone Permutation and Number Invariant Multi-channel Speech Separation](https://mlsanity.com/paper/1910.14104/) (2019)
- [Recognizing Overlapped Speech in Meetings: A Multichannel Separation Approach Using Neural Networks](https://mlsanity.com/paper/1810.03655/) (2018)
- [FaSNet: Low-latency Adaptive Beamforming for Multi-microphone Audio Processing](https://mlsanity.com/paper/1909.13387/) (2019)
- [The CHiME-7 DASR Challenge: Distant Meeting Transcription with Multiple Devices in Diverse Scenarios](https://mlsanity.com/paper/2306.13734/) (2023)
- [Guided Source Separation Meets a Strong ASR Backend: Hitachi/Paderborn University Joint Investigation for Dinner Party ASR](https://mlsanity.com/paper/1905.12230/) (2019)
- [Simulating realistic speech overlaps improves multi-talker ASR](https://mlsanity.com/paper/2210.15715/) (2022)
- [Investigation of Practical Aspects of Single Channel Speech Separation for ASR](https://mlsanity.com/paper/2107.01922/) (2021)
- [VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognition](https://mlsanity.com/paper/2209.04974/) (2022)
- [Towards Low-distortion Multi-channel Speech Enhancement: The ESPNet-SE Submission to The L3DAS22 Challenge](https://mlsanity.com/paper/2202.12298/) (2022)
- [MFCCA:Multi-Frame Cross-Channel attention for multi-speaker ASR in Multi-party meeting scenario](https://mlsanity.com/paper/2210.05265/) (2022)
- [End-to-end Multichannel Speaker-Attributed ASR: Speaker Guided Decoder and Input Feature Analysis](https://mlsanity.com/paper/2310.10106/) (2023)
- [How to Leverage DNN-based speech enhancement for multi-channel speaker verification?](https://mlsanity.com/paper/2210.08834/) (2022)
- [Multi-Talker MVDR Beamforming Based on Extended Complex Gaussian Mixture Model](https://mlsanity.com/paper/1910.07753/) (2019)

## Abstract (arXiv)

Distant-microphone meeting transcription is a challenging task. State-of-the-art end-to-end speaker-attributed automatic speech recognition (SA-ASR) architectures lack a multichannel noise and reverberation reduction front-end, which limits their performance. In this paper, we introduce a joint beamforming and SA-ASR approach for real meeting transcription. We first describe a data alignment and augmentation method to pretrain a neural beamformer on real meeting data. We then compare fixed, hybrid, and fully neural beamformers as front-ends to the SA-ASR model. Finally, we jointly optimize the fully neural beamformer and the SA-ASR model. Experiments on the real AMI corpus show that, while state-of-the-art multi-frame cross-channel attention based channel fusion fails to improve ASR performance, fine-tuning SA-ASR on the fixed beamformer's output and jointly fine-tuning SA-ASR with the neural beamformer reduce the word error rate by 8% and 9% relative, respectively.