# A practical two-stage training strategy for multi-stream end-to-end speech recognition

arXiv: https://arxiv.org/abs/1910.10671 · HTML: https://mlsanity.com/paper/1910.10671/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2019.

## Built on by (most cited first, 3 of 3)

- [Dual-Encoder Architecture with Encoder Selection for Joint Close-Talk and Far-Talk Speech Recognition](https://mlsanity.com/paper/2109.08744/) (2021)
- [Two-Stage Augmentation and Adaptive CTC Fusion for Improved Robustness of Multi-Stream End-to-End ASR](https://mlsanity.com/paper/2102.03055/) (2021)
- [Scaling sparsemax based channel selection for speech recognition with ad-hoc microphone arrays](https://mlsanity.com/paper/2103.15305/) (2021)

## Abstract (arXiv)

The multi-stream paradigm of audio processing, in which several sources are simultaneously considered, has been an active research area for information fusion. Our previous study offered a promising direction within end-to-end automatic speech recognition, where parallel encoders aim to capture diverse information followed by a stream-level fusion based on attention mechanisms to combine the different views. However, with an increasing number of streams resulting in an increasing number of encoders, the previous approach could require substantial memory and massive amounts of parallel data for joint training. In this work, we propose a practical two-stage training scheme. Stage-1 is to train a Universal Feature Extractor (UFE), where encoder outputs are produced from a single-stream model trained with all data. Stage-2 formulates a multi-stream scheme intending to solely train the attention fusion module using the UFE features and pretrained components from Stage-1. Experiments have been conducted on two datasets, DIRHA and AMI, as a multi-stream scenario. Compared with our previous method, this strategy achieves relative word error rate reductions of 8.2--32.4%, while consistently outperforming several conventional combination methods.