# Multistream CNN for Robust Acoustic Modeling

arXiv: https://arxiv.org/abs/2005.10470 · HTML: https://mlsanity.com/paper/2005.10470/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2020.

## Built on by (most cited first, 13 of 13)

- [Leveraging Pre-trained Language Model for Speech Sentiment Analysis](https://mlsanity.com/paper/2106.06598/) (2021)
- [MSTRE-Net: Multistreaming Acoustic Modeling for Automatic Lyrics Transcription](https://mlsanity.com/paper/2108.02625/) (2021)
- [SRU++: Pioneering Fast Recurrence with Attention for Speech Recognition](https://mlsanity.com/paper/2110.05571/) (2021)
- [Advancing Stuttering Detection via Data Augmentation, Class-Balanced Loss and Multi-Contextual Deep Learning](https://mlsanity.com/paper/2302.11343/) (2023)
- [Advancing CTC-CRF Based End-to-End Speech Recognition with Wordpieces and Conformers](https://mlsanity.com/paper/2107.03007/) (2021)
- [Deformable TDNN with adaptive receptive fields for speech recognition](https://mlsanity.com/paper/2104.14791/) (2021)
- [Multi-Convformer: Extending Conformer with Multiple Convolution Kernels](https://mlsanity.com/paper/2407.03718/) (2024)
- [HydraFormer: One Encoder For All Subsampling Rates](https://mlsanity.com/paper/2408.04325/) (2024)
- [CUPE: Contextless Universal Phoneme Encoder for Language-Agnostic Speech Processing](https://mlsanity.com/paper/2508.15316/) (2025)
- [DEFORMER: Coupling Deformed Localized Patterns with Global Context for Robust End-to-end Speech Recognition](https://mlsanity.com/paper/2207.01732/) (2022)
- [ApproBiVT: Lead ASR Models to Generalize Better Using Approximated Bias-Variance Tradeoff Guided Early Stopping and Checkpoint Averaging](https://mlsanity.com/paper/2308.02870/) (2023)
- [The North System for Formosa Speech Recognition Challenge 2023](https://mlsanity.com/paper/2310.03443/) (2023)
- [Speech Representation Learning Revisited: The Necessity of Separate Learnable Parameters and Robust Data Augmentation](https://mlsanity.com/paper/2408.10557/) (2024)

## Abstract (arXiv)

This paper proposes multistream CNN, a novel neural network architecture for robust acoustic modeling in speech recognition tasks. The proposed architecture processes input speech with diverse temporal resolutions by applying different dilation rates to convolutional neural networks across multiple streams to achieve the robustness. The dilation rates are selected from the multiples of a sub-sampling rate of 3 frames. Each stream stacks TDNN-F layers (a variant of 1D CNN), and output embedding vectors from the streams are concatenated then projected to the final layer. We validate the effectiveness of the proposed multistream CNN architecture by showing consistent improvements against Kaldi's best TDNN-F model across various data sets. Multistream CNN improves the WER of the test-other set in the LibriSpeech corpus by 12% (relative). On custom data from ASAPP's production ASR system for a contact center, it records a relative WER improvement of 11% for customer channel audio to prove its robustness to data in the wild. In terms of real-time factor, multistream CNN outperforms the baseline TDNN-F by 15%, which also suggests its practicality on production systems. When combined with self-attentive SRU LM rescoring, multistream CNN contributes for ASAPP to achieve the best WER of 1.75% on test-clean in LibriSpeech.