# Raw Waveform Encoder with Multi-Scale Globally Attentive Locally Recurrent Networks for End-to-End Speech Recognition

arXiv: https://arxiv.org/abs/2106.04275 · HTML: https://mlsanity.com/paper/2106.04275/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale](https://mlsanity.com/paper/1808.10583/) (2018)
- [Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer](https://mlsanity.com/paper/1801.00841/) (2018)
- [Improving RNN Transducer Modeling for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.12415/) (2019)
- [Two-Pass End-to-End Speech Recognition](https://mlsanity.com/paper/1908.10992/) (2019)
- [Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1610.09975/) (2016)
- [Learning Filterbanks from Raw Speech for Phone Recognition](https://mlsanity.com/paper/1711.01161/) (2017)
- [Minimum Bayes Risk Training of RNN-Transducer for End-to-End Speech Recognition](https://mlsanity.com/paper/1911.12487/) (2019)
- [Echo State Speech Recognition](https://mlsanity.com/paper/2102.09114/) (2021)
- [End-to-End Speech Recognition From the Raw Waveform](https://mlsanity.com/paper/1806.07098/) (2018)
- [Sandglasset: A Light Multi-Granularity Self-attentive Network For Time-Domain Speech Separation](https://mlsanity.com/paper/2103.00819/) (2021)
- [Learning Multiscale Features Directly From Waveforms](https://mlsanity.com/paper/1603.09509/) (2016)
- [Exploring RNN-Transducer for Chinese Speech Recognition](https://mlsanity.com/paper/1811.05097/) (2018)
- [Effective Low-Cost Time-Domain Audio Separation Using Globally Attentive Locally Recurrent Networks](https://mlsanity.com/paper/2101.05014/) (2021)
- [Speech and Speaker Recognition from Raw Waveform with SincNet](https://mlsanity.com/paper/1812.05920/) (2018)
- [Tune-In: Training Under Negative Environments with Interference for Attention Networks Simulating Cocktail Party Effect](https://mlsanity.com/paper/2103.01461/) (2021)

## Built on by (most cited first, 2 of 2)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Revisiting End-to-End Speech-to-Text Translation From Scratch](https://mlsanity.com/paper/2206.04571/) (2022)

## Abstract (arXiv)

End-to-end speech recognition generally uses hand-engineered acoustic features as input and excludes the feature extraction module from its joint optimization. To extract learnable and adaptive features and mitigate information loss, we propose a new encoder that adopts globally attentive locally recurrent (GALR) networks and directly takes raw waveform as input. We observe improved ASR performance and robustness by applying GALR on different window lengths to aggregate fine-grain temporal information into multi-scale acoustic features. Experiments are conducted on a benchmark dataset AISHELL-2 and two large-scale Mandarin speech corpus of 5,000 hours and 21,000 hours. With faster speed and comparable model size, our proposed multi-scale GALR waveform encoder achieved consistent character error rate reductions (CERRs) from 7.9% to 28.1% relative over strong baselines, including Conformer and TDNN-Conformer. In particular, our approach demonstrated notable robustness than the traditional handcrafted features and outperformed the baseline MFCC-based TDNN-Conformer model by a 15.2% CERR on a music-mixed real-world speech test set.