# Echo State Speech Recognition

arXiv: https://arxiv.org/abs/2102.09114 · HTML: https://mlsanity.com/paper/2102.09114/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 23 references that are in the map.
- First posted on arXiv 18 Feb 2021.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency](https://mlsanity.com/paper/2003.12710/) (2020)
- [Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer](https://mlsanity.com/paper/1801.00841/) (2018)
- [Comparison between DeepESNs and gated RNNs on multivariate time-series prediction](https://mlsanity.com/paper/1812.11527/) (2018)
- [Richness of Deep Echo State Network Dynamics](https://mlsanity.com/paper/1903.05174/) (2019)

## Built on by (most cited first, 11 of 11)

- [Efficient Sequence Transduction by Jointly Predicting Tokens and Durations](https://mlsanity.com/paper/2304.06795/) (2023)
- [Tied & Reduced RNN-T Decoder](https://mlsanity.com/paper/2109.07513/) (2021)
- [Efficient and Private Federated Learning with Partially Trainable Networks](https://mlsanity.com/paper/2110.03450/) (2021)
- [CIF-T: A Novel CIF-based Transducer Architecture for Automatic Speech Recognition](https://mlsanity.com/paper/2307.14132/) (2023)
- [Replay to Remember: Continual Layer-Specific Fine-tuning for German Speech Recognition](https://mlsanity.com/paper/2307.07280/) (2023)
- [Neural Graph Revealers](https://mlsanity.com/paper/2302.13582/) (2023)
- [Compressing 1D Time-Channel Separable Convolutions using Sparse Random Ternary Matrices](https://mlsanity.com/paper/2103.17142/) (2021)
- [Raw Waveform Encoder with Multi-Scale Globally Attentive Locally Recurrent Networks for End-to-End Speech Recognition](https://mlsanity.com/paper/2106.04275/) (2021)
- [Exploring Heterogeneous Characteristics of Layers in ASR Models for More Efficient Training](https://mlsanity.com/paper/2110.04267/) (2021)
- [On the Prediction Network Architecture in RNN-T for ASR](https://mlsanity.com/paper/2206.14618/) (2022)
- [Variability of echo state network prediction horizon for partially observed dynamical systems](https://mlsanity.com/paper/2306.10797/) (2023)

## Abstract (arXiv)

We propose automatic speech recognition (ASR) models inspired by echo state network (ESN), in which a subset of recurrent neural networks (RNN) layers in the models are randomly initialized and untrained. Our study focuses on RNN-T and Conformer models, and we show that model quality does not drop even when the decoder is fully randomized. Furthermore, such models can be trained more efficiently as the decoders do not require to be updated. By contrast, randomizing encoders hurts model quality, indicating that optimizing encoders and learn proper representations for acoustic inputs are more vital for speech recognition. Overall, we challenge the common practice of training ASR models for all components, and demonstrate that ESN-based models can perform equally well but enable more efficient training and storage than fully-trainable counterparts.