mlsanityData to 14 Oct 2026

Pay Less Attention with Lightweight and Dynamic Convolutions

arXiv 1901.10430 ↗First posted 29 Jan 2019cs.CLcited by 211

Not on the map. Why?

Abstract (arXiv)

Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current time step. In this paper, we show that a very lightweight convolution can perform competitively to the best reported self-attention results. Next, we introduce dynamic convolutions which are simpler and more efficient than self-attention. We predict separate convolution kernels based solely on the current time-step in order to determine the importance of context elements. The number of operations required by this approach scales linearly in the input length, whereas self-attention is quadratic. Experiments on large-scale machine translation, language modeling and abstractive summarization show that dynamic convolutions improve over strong self-attention models. On the WMT'14 English-German test set dynamic convolutions achieve a new state of the art of 29.7 BLEU.

Key facts. Cited by 211 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 85; 2022: 51; 2023: 41; 2024: 16; 2025: 10; 2026: 8.

211 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  2. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  3. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  4. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  5. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  6. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  7. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  8. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  9. 2021A Survey of Transformerscited by 401
  10. 2021Pay Attention to MLPscited by 368
  11. 2021Random Feature Attentioncited by 354
  12. 2021Fast Convergence of DETR with Spatially Modulated Co-Attentioncited by 176
38 more
  1. 2021VOLO: Vision Outlooker for Visual Recognitioncited by 165
  2. 2022Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNscited by 155
  3. 2024MambaOut: Do We Really Need Mamba for Vision?cited by 143
  4. 2022MetaFormer Baselines for Visioncited by 132
  5. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 130
  6. 2021OW-DETR: Open-world Detection Transformercited by 129
  7. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 124
  8. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 113
  9. 2021An Attention Free Transformercited by 112
  10. 2021Learning Graph Structures with Transformer for Multivariate Time Series Anomaly Detection in IoTcited by 104
  11. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  12. 2021Finetuning Pretrained Transformers into RNNscited by 96
  13. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  14. 2021Mixed Transformer U-Net For Medical Image Segmentationcited by 95
  15. 2021Survey of Low-Resource Machine Translationcited by 92
  16. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 92
  17. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  18. 2022SeedFormer: Patch Seeds based Point Cloud Completion with Upsample Transformercited by 83
  19. 2021How to Train BERT with an Academic Budgetcited by 80
  20. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  21. 2021Direct Multi-view Multi-person 3D Pose Estimationcited by 64
  22. 2021On the Connection between Local Attention and Dynamic Depth-wise Convolutioncited by 62
  23. 2021Poolingformer: Long Document Modeling with Pooling Attentioncited by 60
  24. 2021Adaptive Graph Convolution for Point Cloud Analysiscited by 60
  25. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  26. 2022Mixture of Attention Heads: Selecting Attention Heads Per Tokencited by 60
  27. 2021NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Searchcited by 56
  28. 2021Spatio-temporal Relation Modeling for Few-shot Action Recognitioncited by 53
  29. 2021A Survey on Dialogue Summarization: Recent Advances and New Frontierscited by 51
  30. 2024Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applicationscited by 50
  31. 2021Learning Language Specific Sub-network for Multilingual Machine Translationcited by 48
  32. 2021Towards Building ASR Systems for the Next Billion Userscited by 47
  33. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  34. 2024OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentationcited by 44
  35. 2021Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformerscited by 43
  36. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 41
  37. 2023You Only Segment Once: Towards Real-Time Panoptic Segmentationcited by 40
  38. 2021Unified Graph Structured Models for Video Understandingcited by 38

50 most cited of 211 All 211 (JSON)