# DFSMN-SAN with Persistent Memory Model for Automatic Speech Recognition

arXiv: https://arxiv.org/abs/1910.13282 · HTML: https://mlsanity.com/paper/1910.13282/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Oct 2019.

## Built on by (most cited first, 4 of 4)

- [SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts](https://mlsanity.com/paper/2105.03036/) (2021)
- [Efficient Transformer for Direct Speech Translation](https://mlsanity.com/paper/2107.03069/) (2021)
- [A Unified Speaker Adaptation Approach for ASR](https://mlsanity.com/paper/2110.08545/) (2021)
- [Referee: Towards reference-free cross-speaker style transfer with low-quality data for expressive speech synthesis](https://mlsanity.com/paper/2109.03439/) (2021)

## Abstract (arXiv)

Self-attention networks (SAN) have been introduced into automatic speech recognition (ASR) and achieved state-of-the-art performance owing to its superior ability in capturing long term dependency. One of the key ingredients is the self-attention mechanism which can be effectively performed on the whole utterance level. In this paper, we try to investigate whether even more information beyond the whole utterance level can be exploited and beneficial. We propose to apply self-attention layer with augmented memory to ASR. Specifically, we first propose a variant model architecture which combines deep feed-forward sequential memory network (DFSMN) with self-attention layers to form a better baseline model compared with a purely self-attention network. Then, we propose and compare two kinds of additional memory structures added into self-attention layers. Experiments on large-scale LVCSR tasks show that on four individual test sets, the DFSMN-SAN architecture outperforms vanilla SAN encoder by 5% relatively in character error rate (CER). More importantly, the additional memory structure provides further 5% to 11% relative improvement in CER.