mlsanityData to 7 Oct 2026

Language Modeling with Gated Convolutional Networks

arXiv 1612.08083 ↗First posted 23 Dec 2016cs.CLcited by 688

Not on the map. Why?

Cited by 24 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The pre-dominant approach to language modeling to date is based on recurrent neural networks. Their success on this task is often linked to their ability to capture unbounded context. In this paper we develop a finite context approach through stacked convolutions, which can be more efficient since they allow parallelization over sequential tokens. We propose a novel simplified gating mechanism that outperforms Oord et al (2016) and investigate the impact of key architectural decisions. The proposed approach achieves state-of-the-art on the WikiText-103 benchmark, even though it features long-term dependencies, as well as competitive results on the Google Billion Words benchmark. Our model reduces the latency to score a sentence by an order of magnitude compared to a recurrent baseline. To our knowledge, this is the first time a non-recurrent approach is competitive with strong recurrent models on these large scale language tasks.

Key facts. Cited by 688 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 1; 2021: 99; 2022: 88; 2023: 111; 2024: 155; 2025: 132; 2026: 101.

688 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  5. 2024Qwen2 Technical Reportcited by 2,076
  6. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,618
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  8. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  9. 2022Simple Baselines for Image Restorationcited by 701
  10. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  11. 2023A Comprehensive Overview of Large Language Modelscited by 639
  12. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
38 more
  1. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  2. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  3. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  4. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  5. 2021A Survey of Transformerscited by 399
  6. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  7. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  8. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  9. 2021Pay Attention to MLPscited by 367
  10. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 331
  11. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  12. 2021Spectral Temporal Graph Neural Network for Multivariate Time-series Forecastingcited by 298
  13. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  14. 2022Transformer for Partial Differential Equations' Operator Learningcited by 270
  15. 2022Long Range Language Modeling via Gated State Spacescited by 265
  16. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  17. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  18. 2022Transformer Quality in Linear Timecited by 189
  19. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  20. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  21. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 171
  22. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  23. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  24. 2025Qwen2.5-1M Technical Reportcited by 158
  25. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  26. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  27. 2024MambaOut: Do We Really Need Mamba for Vision?cited by 142
  28. 2021Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesiscited by 140
  29. 2023Spatio-Temporal Graph Neural Networks for Predictive Learning in Urban Computing: A Surveycited by 140
  30. 2023DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomecited by 139
  31. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  32. 2022Decoding speech perception from non-invasive brain recordingscited by 129
  33. 2023End-to-End Speech Recognition: A Surveycited by 127
  34. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 117
  35. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 117
  36. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  37. 2023Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documentscited by 103
  38. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101

50 most cited of 688 All 688 (JSON)