# End-to-End Dereverberation, Beamforming, and Speech Recognition with Improved Numerical Stability and Advanced Frontend

arXiv: https://arxiv.org/abs/2102.11525 · HTML: https://mlsanity.com/paper/2102.11525/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Feb 2021.

## Built on by (most cited first, 12 of 12)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [VarArray: Array-Geometry-Agnostic Continuous Speech Separation](https://mlsanity.com/paper/2110.05745/) (2021)
- [ESPnet-SE++: Speech Enhancement for Robust Speech Recognition, Translation, and Understanding](https://mlsanity.com/paper/2207.09514/) (2022)
- [VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognition](https://mlsanity.com/paper/2209.04974/) (2022)
- [MIMO Self-attentive RNN Beamformer for Multi-speaker Speech Separation](https://mlsanity.com/paper/2104.08450/) (2021)
- [Self-Attention Channel Combinator Frontend for End-to-End Multichannel Far-field Speech Recognition](https://mlsanity.com/paper/2109.04783/) (2021)
- [Joint Modelling of Spoken Language Understanding Tasks with Integrated Dialog History](https://mlsanity.com/paper/2305.00926/) (2023)
- [A Time-domain Real-valued Generalized Wiener Filter for Multi-channel Neural Separation Systems](https://mlsanity.com/paper/2112.03533/) (2021)
- [End-to-End Multi-speaker ASR with Independent Vector Analysis](https://mlsanity.com/paper/2204.00218/) (2022)
- [Customizable End-to-end Optimization of Online Neural Network-supported Dereverberation for Hearing Devices](https://mlsanity.com/paper/2204.02694/) (2022)
- [A neural network-supported two-stage algorithm for lightweight dereverberation on hearing devices](https://mlsanity.com/paper/2204.02978/) (2022)
- [Factorised Speaker-environment Adaptive Training of Conformer Speech Recognition Systems](https://mlsanity.com/paper/2306.14608/) (2023)

## Abstract (arXiv)

Recently, the end-to-end approach has been successfully applied to multi-speaker speech separation and recognition in both single-channel and multichannel conditions. However, severe performance degradation is still observed in the reverberant and noisy scenarios, and there is still a large performance gap between anechoic and reverberant conditions. In this work, we focus on the multichannel multi-speaker reverberant condition, and propose to extend our previous framework for end-to-end dereverberation, beamforming, and speech recognition with improved numerical stability and advanced frontend subnetworks including voice activity detection like masks. The techniques significantly stabilize the end-to-end training process. The experiments on the spatialized wsj1-2mix corpus show that the proposed system achieves about 35% WER relative reduction compared to our conventional multi-channel E2E ASR system, and also obtains decent speech dereverberation and separation performance (SDR=12.5 dB) in the reverberant multi-speaker condition while trained only with the ASR criterion.