# Dual-Encoder Architecture with Encoder Selection for Joint Close-Talk and Far-Talk Speech Recognition

arXiv: https://arxiv.org/abs/2109.08744 · HTML: https://mlsanity.com/paper/2109.08744/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 40 references that are in the map.
- First posted on arXiv 17 Sep 2021.

## Builds on (most cited first, 24 of 24)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Cascaded encoders for unifying streaming and non-streaming ASR](https://mlsanity.com/paper/2010.14606/) (2020)
- [End-to-End Multi-speaker Speech Recognition with Transformer](https://mlsanity.com/paper/2002.03921/) (2020)
- [Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboard](https://mlsanity.com/paper/2001.07263/) (2020)
- [Far-Field Automatic Speech Recognition](https://mlsanity.com/paper/2009.09395/) (2020)
- [Semi-Supervised Learning with Data Augmentation for End-to-End ASR](https://mlsanity.com/paper/2007.13876/) (2020)
- [Attentive Convolutional Neural Network based Speech Emotion Recognition: A Study on the Impact of Input Features, Signal Length, and Acted Speech](https://mlsanity.com/paper/1706.00612/) (2017)
- [End-to-End Multi-Channel Transformer for Speech Recognition](https://mlsanity.com/paper/2102.03951/) (2021)
- [Exploring attention mechanism for acoustic-based classification of speech utterances into system-directed and non-system-directed](https://mlsanity.com/paper/1902.00570/) (2019)
- [Frequency Domain Multi-channel Acoustic Modeling for Distant Speech Recognition](https://mlsanity.com/paper/1903.05299/) (2019)
- [Listen, Attend, Spell and Adapt: Speaker Adapted Sequence-to-Sequence ASR](https://mlsanity.com/paper/1907.04916/) (2019)
- [Multi-Stream End-to-End Speech Recognition](https://mlsanity.com/paper/1906.08041/) (2019)
- [Bifocal Neural ASR: Exploiting Keyword Spotting for Inference Optimization](https://mlsanity.com/paper/2108.01704/) (2021)
- [Multi-Geometry Spatial Acoustic Modeling for Distant Speech Recognition](https://mlsanity.com/paper/1903.06539/) (2019)
- [Exploring End-to-End Multi-channel ASR with Bias Information for Meeting Transcription](https://mlsanity.com/paper/2011.03110/) (2020)
- [Stream attention-based multi-array end-to-end speech recognition](https://mlsanity.com/paper/1811.04903/) (2018)
- [A Study of Enhancement, Augmentation, and Autoencoder Methods for Domain Adaptation in Distant Speech Recognition](https://mlsanity.com/paper/1806.04841/) (2018)
- [A practical two-stage training strategy for multi-stream end-to-end speech recognition](https://mlsanity.com/paper/1910.10671/) (2019)
- [Mitigating the Impact of Speech Recognition Errors on Chatbot using Sequence-to-Sequence Model](https://mlsanity.com/paper/1709.07862/) (2017)
- [Two-Stage Augmentation and Adaptive CTC Fusion for Improved Robustness of Multi-Stream End-to-End ASR](https://mlsanity.com/paper/2102.03055/) (2021)

## Built on by (most cited first, 4 of 4)

- [Conformer with dual-mode chunked attention for joint online and offline ASR](https://mlsanity.com/paper/2206.11157/) (2022)
- [I3D: Transformer architectures with input-dependent dynamic depth for speech recognition](https://mlsanity.com/paper/2303.07624/) (2023)
- [Compute Cost Amortized Transformer for Streaming ASR](https://mlsanity.com/paper/2207.02393/) (2022)
- [Challenges and Opportunities in Multi-device Speech Processing](https://mlsanity.com/paper/2206.15432/) (2022)

## Abstract (arXiv)

In this paper, we propose a dual-encoder ASR architecture for joint modeling of close-talk (CT) and far-talk (FT) speech, in order to combine the advantages of CT and FT devices for better accuracy. The key idea is to add an encoder selection network to choose the optimal input source (CT or FT) and the corresponding encoder. We use a single-channel encoder for CT speech and a multi-channel encoder with Spatial Filtering neural beamforming for FT speech, which are jointly trained with the encoder selection. We validate our approach on both attention-based and RNN Transducer end-to-end ASR systems. The experiments are done with conversational speech from a medical use case, which is recorded simultaneously with a CT device and a microphone array. Our results show that the proposed dual-encoder architecture obtains up to 9% relative WER reduction when using both CT and FT input, compared to the best single-encoder system trained and tested in matched condition.