Data to 14 Oct 2026

Quasi-Recurrent Neural Networks

arXiv 1611.01576 ↗First posted 5 Nov 2016cs.NE cs.AI cs.CLcited by 87

Not on the map. Why?

Abstract (arXiv)

Recurrent neural networks are a powerful tool for modeling sequential data, but the dependence of each timestep's computation on the previous timestep's output limits parallelism and makes RNNs unwieldy for very long sequences. We introduce quasi-recurrent neural networks (QRNNs), an approach to neural sequence modeling that alternates convolutional layers, which apply in parallel across timesteps, and a minimalist recurrent pooling function that applies in parallel across channels. Despite lacking trainable recurrent layers, stacked QRNNs have better predictive accuracy than stacked LSTMs of the same hidden size. Due to their increased parallelism, they are up to 16 times faster at train and test time. Experiments on language modeling, sentiment classification, and character-level neural machine translation demonstrate these advantages and underline the viability of QRNNs as a basic building block for a variety of sequence tasks.

Key facts. Cited by 87 papers in the map (data through 14 Oct 2026). First posted on arXiv 5 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 31; 2022: 10; 2023: 14; 2024: 15; 2025: 11; 2026: 5.

87 papers.

Sort:

By area:

2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
2022Simplified State Space Layers for Sequence Modelingcited by 788
2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 216
2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 154
2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 74
2021Stabilizing Equilibrium Models by Jacobian Regularizationcited by 72
2022Audio Self-supervised Learning: A Surveycited by 65
2021Nondeterminism and Instability in Neural Network Optimizationcited by 48

38 more

2024The Expressive Capacity of State Space Models: A Formal Language Perspectivecited by 46
2021A Dynamic Spatial-temporal Attention Network for Early Anticipation of Traffic Accidentscited by 42
2021Evaluating Saliency Methods for Neural Language Modelscited by 37
2021When Attention Meets Fast Recurrence: Training Language Models with Reduced Computecited by 34
2021Dual-side Sparse Tensor Corecited by 29
2022Efficient recurrent architectures through activity sparsity and sparse back-propagation through timecited by 26
2024MetaLA: Unified Optimal Linear Approximation to Softmax Attention Mapcited by 21
2023Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensionscited by 20
2022pNLP-Mixer: an Efficient all-MLP Architecture for Languagecited by 19
2023Exploring the Promise and Limits of Real-Time Recurrent Learningcited by 19
2025StateSpaceDiffuser: Bringing Long Context to Diffusion World Modelscited by 19
2021Multi-Task Self-Supervised Pre-Training for Music Classificationcited by 18
2021Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNNcited by 16
2023Hybrid Spectral Denoising Transformer with Guided Attentioncited by 16
2021Enhancing Speaking Styles in Conversational Text-to-Speech Synthesis with Graph-based Multi-modal Context Modelingcited by 15
2023Illuminati: Towards Explaining Graph Neural Networks for Cybersecurity Analysiscited by 14
2021Capitalization and Punctuation Restoration: a Surveycited by 13
2025Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformerscited by 12
2023Unsupervised Deep Learning for IoT Time Seriescited by 11
2021Distilling Large Language Models into Tiny and Effective Students using pQRNNcited by 10
2021Low-Resource Language Modelling of South African Languagescited by 10
2023The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Computecited by 10
2025Sequential-Parallel Duality in Prefix Scannable Modelscited by 8
2025Fast weight programming and linear transformers: from machine learning to neurobiologycited by 8
2026Why Are Linear RNNs More Parallelizable?cited by 8
2021RIANN -- A Robust Neural Network Outperforms Attitude Estimation Filterscited by 6
2022The Efficacy of Self-Supervised Speech Models for Audio Representationscited by 6
2023Koopman Invertible Autoencoder: Leveraging Forward and Backward Dynamics for Temporal Modelingcited by 6
2024Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyphcited by 6
2021Doping: A technique for efficient compression of LSTM models using sparse structured additive matricescited by 5
2021Representation based meta-learning for few-shot spoken intent recognitioncited by 5
2021Rumour Detection via Zero-shot Cross-lingual Transfer Learningcited by 5
2023Fusing Structure from Motion and Simulation-Augmented Pose Regression from Optical Flow for Challenging Indoor Environmentscited by 5
20213D Convolutional Neural Networks for Ultrasound-Based Silent Speech Interfacescited by 4
2022Simple Recurrence Improves Masked Language Modelscited by 4
2022FedorAS: Federated Architecture Search under system heterogeneitycited by 4
2022Streaming Intended Query Detection using E2E Modeling for Continued Conversationcited by 3
2024Streaming Detection of Queried Event Startcited by 3

50 most cited of 87 All 87 (JSON)