# Feedforward Sequential Memory Networks: A New Structure to Learn Long-term Dependency

arXiv: https://arxiv.org/abs/1512.08301 · HTML: https://mlsanity.com/paper/1512.08301/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Dec 2015.

## Built on by (most cited first, 10 of 10)

- [BiBench: Benchmarking and Analyzing Network Binarization](https://mlsanity.com/paper/2301.11233/) (2023)
- [ATCSpeechNet: A multilingual end-to-end speech recognition framework for air traffic control systems](https://mlsanity.com/paper/2102.08535/) (2021)
- [BiFSMN: Binary Neural Network for Keyword Spotting](https://mlsanity.com/paper/2202.06483/) (2022)
- [A Comparative Study on Speaker-attributed Automatic Speech Recognition in Multi-party Meetings](https://mlsanity.com/paper/2203.16834/) (2022)
- [Improving CTC-based ASR Models with Gated Interlayer Collaboration](https://mlsanity.com/paper/2205.12462/) (2022)
- [BiFSMNv2: Pushing Binary Neural Networks for Keyword Spotting to Real-Network Performance](https://mlsanity.com/paper/2211.06987/) (2022)
- [Speaker Embedding-aware Neural Diarization: an Efficient Framework for Overlapping Speech Diarization in Meeting Scenarios](https://mlsanity.com/paper/2203.09767/) (2022)
- [Combining Slow and Fast: Complementary Filtering for Dynamics Learning](https://mlsanity.com/paper/2302.13754/) (2023)
- [BiND: A Neural Discriminator-Decoder for Accurate Bimanual Trajectory Prediction in Brain-Computer Interfaces](https://mlsanity.com/paper/2509.03521/) (2025)
- [UAF: A Unified Audio Front-end LLM for Full-Duplex Speech Interaction](https://mlsanity.com/paper/2604.19221/) (2026)

## Abstract (arXiv)

In this paper, we propose a novel neural network structure, namely feedforward sequential memory networks (FSMN), to model long-term dependency in time series without using recurrent feedback. The proposed FSMN is a standard fully-connected feedforward neural network equipped with some learnable memory blocks in its hidden layers. The memory blocks use a tapped-delay line structure to encode the long context information into a fixed-size representation as short-term memory mechanism. We have evaluated the proposed FSMNs in several standard benchmark tasks, including speech recognition and language modelling. Experimental results have shown FSMNs significantly outperform the conventional recurrent neural networks (RNN), including LSTMs, in modeling sequential signals like speech or language. Moreover, FSMNs can be learned much more reliably and faster than RNNs or LSTMs due to the inherent non-recurrent model structure.