mlsanityData to 7 Oct 2026

A Primal-Dual Framework for Transformers and Neural Networks

arXiv 2406.13781 ↗First posted 19 Jun 2024cs.LG cs.AI cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Self-attention is key to the remarkable success of transformers in sequence modeling tasks including many applications in natural language processing and computer vision. Like neural network layers, these attention mechanisms are often developed by heuristics and experience. To provide a principled framework for constructing attention layers in transformers, we show that the self-attention corresponds to the support vector expansion derived from a support vector regression problem, whose primal formulation has the form of a neural network layer. Using our framework, we derive popular attention layers used in practice and propose two new attentions: 1) the Batch Normalized Attention (Attention-BN) derived from the batch normalization layer and 2) the Attention with Scaled Head (Attention-SH) derived from using less training data to fit the SVR model. We empirically demonstrate the advantages of the Attention-BN and Attention-SH in reducing head redundancy, increasing the model's accuracy, and improving the model's efficiency in a variety of practical applications including image and time-series classification.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 74 of its 98 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

74 of 98 references.

References · 74

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  8. 2017Non-local Neural Networkscited by 2,290
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  10. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  12. 2020Reformer: The Efficient Transformercited by 1,658
  13. 2021ViViT: A Video Vision Transformercited by 1,610
  14. 2020Rethinking Attention with Performerscited by 1,584
  15. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  16. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  17. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  18. 2020Point Transformercited by 1,462
  19. 2018Self-Attention with Relative Position Representationscited by 1,112
  20. 2021Video Swin Transformercited by 1,090
  21. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  22. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  23. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  24. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  25. 2018Image Transformercited by 770
  26. 2020PCT: Point cloud transformercited by 730
  27. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  28. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  29. 2019Designing and Interpreting Probes with Control Taskscited by 608
  30. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  31. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  32. 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
  33. 2020A Transformer-based Framework for Multivariate Time Series Representation Learningcited by 449
  34. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  35. 2019Axial Attention in Multidimensional Transformerscited by 350
  36. 2021Random Feature Attentioncited by 350
  37. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  38. 2021Choose a Transformer: Fourier or Galerkincited by 333

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Tight Clusters Make Specialized Expertscited by 8
  2. 2025Transformer Meets Twicing: Harnessing Unattended Residual Informationcited by 3
  3. 2025Compact Vision Transformer by Reduction of Kernel Complexitycited by 0
  4. 2026VideoSEMA: a scalable and efficient Mamba-like attention for video understandingcited by 0