# Deep-FSMN for Large Vocabulary Continuous Speech Recognition

arXiv: https://arxiv.org/abs/1803.05030 · HTML: https://mlsanity.com/paper/1803.05030/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2018.

## Built on by (most cited first, 28 of 28)

- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [FunASR: A Fundamental End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2305.11013/) (2023)
- [One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Era](https://mlsanity.com/paper/2304.06488/) (2023)
- [SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts](https://mlsanity.com/paper/2105.03036/) (2021)
- [DiffSVC: A Diffusion Probabilistic Model for Singing Voice Conversion](https://mlsanity.com/paper/2105.13871/) (2021)
- [LongCat-Flash-Omni Technical Report](https://mlsanity.com/paper/2511.00279/) (2025)
- [BiBench: Benchmarking and Analyzing Network Binarization](https://mlsanity.com/paper/2301.11233/) (2023)
- [Interactive Conversational Head Generation](https://mlsanity.com/paper/2307.02090/) (2023)
- [Towards High-fidelity Singing Voice Conversion with Acoustic Reference and Contrastive Predictive Coding](https://mlsanity.com/paper/2110.04754/) (2021)
- [Cross-Modal ASR Post-Processing System for Error Correction and Utterance Rejection](https://mlsanity.com/paper/2201.03313/) (2022)
- [BiFSMN: Binary Neural Network for Keyword Spotting](https://mlsanity.com/paper/2202.06483/) (2022)
- [3D Convolutional Neural Networks for Ultrasound-Based Silent Speech Interfaces](https://mlsanity.com/paper/2104.11532/) (2021)
- [BiFSMNv2: Pushing Binary Neural Networks for Keyword Spotting to Real-Network Performance](https://mlsanity.com/paper/2211.06987/) (2022)
- [Improving Non-native Word-level Pronunciation Scoring with Phone-level Mixup Data Augmentation and Multi-source Information](https://mlsanity.com/paper/2203.01826/) (2022)
- [Integrating Lattice-Free MMI into End-to-End Speech Recognition](https://mlsanity.com/paper/2203.15614/) (2022)
- [Phonetic and Prosody-aware Self-supervised Learning Approach for Non-native Fluency Scoring](https://mlsanity.com/paper/2305.11438/) (2023)
- [Full-text Error Correction for Chinese Speech Recognition with Large Language Model](https://mlsanity.com/paper/2409.07790/) (2024)
- [Speaker Embedding-aware Neural Diarization: an Efficient Framework for Overlapping Speech Diarization in Meeting Scenarios](https://mlsanity.com/paper/2203.09767/) (2022)
- [Internal Language Model Estimation based Adaptive Language Model Fusion for Domain Adaptation](https://mlsanity.com/paper/2211.00968/) (2022)
- [Tencent AVS: A Holistic Ads Video Dataset for Multi-modal Scene Segmentation](https://mlsanity.com/paper/2212.04700/) (2022)
- [Multi-Task Learning based Online Dialogic Instruction Detection with Pre-trained Language Models](https://mlsanity.com/paper/2107.07119/) (2021)
- [ICMC-ASR: The ICASSP 2024 In-Car Multi-Channel Automatic Speech Recognition Challenge](https://mlsanity.com/paper/2401.03473/) (2024)
- [Boosting Tail Neural Network for Realtime Custom Keyword Spotting](https://mlsanity.com/paper/2205.12933/) (2022)
- [DialogID: A Dialogic Instruction Dataset for Improving Teaching Effectiveness in Online Environments](https://mlsanity.com/paper/2206.12034/) (2022)
- [Seewo's Submission to MLC-SLM: Lessons learned from Speech Reasoning Language Models](https://mlsanity.com/paper/2506.13300/) (2025)
- [A Small-footprint Acoustic Echo Cancellation Solution for Mobile Full-Duplex Speech Interactions](https://mlsanity.com/paper/2508.07561/) (2025)
- [UAF: A Unified Audio Front-end LLM for Full-Duplex Speech Interaction](https://mlsanity.com/paper/2604.19221/) (2026)
- [FUTO Swipe: Layout-Agnostic Neural Swipe Decoding](https://mlsanity.com/paper/2606.25247/) (2026)

## Abstract (arXiv)

In this paper, we present an improved feedforward sequential memory networks (FSMN) architecture, namely Deep-FSMN (DFSMN), by introducing skip connections between memory blocks in adjacent layers. These skip connections enable the information flow across different layers and thus alleviate the gradient vanishing problem when building very deep structure. As a result, DFSMN significantly benefits from these skip connections and deep structure. We have compared the performance of DFSMN to BLSTM both with and without lower frame rate (LFR) on several large speech recognition tasks, including English and Mandarin. Experimental results shown that DFSMN can consistently outperform BLSTM with dramatic gain, especially trained with LFR using CD-Phone as modeling units. In the 2000 hours Fisher (FSH) task, the proposed DFSMN can achieve a word error rate of 9.4% by purely using the cross-entropy criterion and decoding with a 3-gram language model, which achieves a 1.5% absolute improvement compared to the BLSTM. In a 20000 hours Mandarin recognition task, the LFR trained DFSMN can achieve more than 20% relative improvement compared to the LFR trained BLSTM. Moreover, we can easily design the lookahead filter order of the memory blocks in DFSMN to control the latency for real-time applications.