# A unified multichannel far-field speech recognition system: combining neural beamforming with attention based end-to-end model

arXiv: https://arxiv.org/abs/2401.02673 · HTML: https://mlsanity.com/paper/2401.02673/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 5 Jan 2024.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [Pyroomacoustics: A Python package for audio room simulations and array processing algorithms](https://mlsanity.com/paper/1710.04196/) (2017)
- [An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation](https://mlsanity.com/paper/2008.09586/) (2020)
- [EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding](https://mlsanity.com/paper/1507.08240/) (2015)
- [End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results](https://mlsanity.com/paper/1412.1602/) (2014)
- [A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition](https://mlsanity.com/paper/1807.10857/) (2018)
- [gpuRIR: A Python Library for Room Impulse Response Simulation with GPU Acceleration](https://mlsanity.com/paper/1810.11359/) (2018)
- [Multichannel End-to-end Speech Recognition](https://mlsanity.com/paper/1703.04783/) (2017)
- [Neural Spatio-Temporal Beamformer for Target Speech Separation](https://mlsanity.com/paper/2005.03889/) (2020)
- [Spatial Attention for Far-field Speech Recognition with Deep Beamforming Neural Networks](https://mlsanity.com/paper/1911.02115/) (2019)

## Abstract (arXiv)

Far-field speech recognition is a challenging task that conventionally uses signal processing beamforming to attack noise and interference problem. But the performance has been found usually limited due to heavy reliance on environmental assumption. In this paper, we propose a unified multichannel far-field speech recognition system that combines the neural beamforming and transformer-based Listen, Spell, Attend (LAS) speech recognition system, which extends the end-to-end speech recognition system further to include speech enhancement. Such framework is then jointly trained to optimize the final objective of interest. Specifically, factored complex linear projection (fCLP) has been adopted to form the neural beamforming. Several pooling strategies to combine look directions are then compared in order to find the optimal approach. Moreover, information of the source direction is also integrated in the beamforming to explore the usefulness of source direction as a prior, which is usually available especially in multi-modality scenario. Experiments on different microphone array geometry are conducted to evaluate the robustness against spacing variance of microphone array. Large in-house databases are used to evaluate the effectiveness of the proposed framework and the proposed method achieve 19.26% improvement when compared with a strong baseline.