# Multi-view Frequency LSTM: An Efficient Frontend for Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2007.00131 · HTML: https://mlsanity.com/paper/2007.00131/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jun 2020.

## Built on by (most cited first, 3 of 3)

- [Do You Listen with One or Two Microphones? A Unified ASR Model for Single and Multi-Channel Audio](https://mlsanity.com/paper/2106.02750/) (2021)
- [Time-Frequency Localization Using Deep Convolutional Maxout Neural Network in Persian Speech Recognition](https://mlsanity.com/paper/2108.03818/) (2021)
- [Multi-View Frequency-Attention Alternative to CNN Frontends for Automatic Speech Recognition](https://mlsanity.com/paper/2306.06954/) (2023)

## Abstract (arXiv)

Acoustic models in real-time speech recognition systems typically stack multiple unidirectional LSTM layers to process the acoustic frames over time. Performance improvements over vanilla LSTM architectures have been reported by prepending a stack of frequency-LSTM (FLSTM) layers to the time LSTM. These FLSTM layers can learn a more robust input feature to the time LSTM layers by modeling time-frequency correlations in the acoustic input signals. A drawback of FLSTM based architectures however is that they operate at a predefined, and tuned, window size and stride, referred to as 'view' in this paper. We present a simple and efficient modification by combining the outputs of multiple FLSTM stacks with different views, into a dimensionality reduced feature representation. The proposed multi-view FLSTM architecture allows to model a wider range of time-frequency correlations compared to an FLSTM model with single view. When trained on 50K hours of English far-field speech data with CTC loss followed by sMBR sequence training, we show that the multi-view FLSTM acoustic model provides relative Word Error Rate (WER) improvements of 3-7% for different speaker and acoustic environment scenarios over an optimized single FLSTM model, while retaining a similar computational footprint.