mlsanityData to 7 Oct 2026

UFO-ViT: High Performance Linear Vision Transformer without Softmax

arXiv 2109.14382 ↗First posted 29 Sep 2021cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

Vision transformers have become one of the most important models for computer vision tasks. Although they outperform prior works, they require heavy computational resources on a scale that is quadratic to N. This is a major drawback of the traditional self-attention (SA) algorithm. Here, we propose the Unit Force Operated Vision Transformer (UFO-ViT), a novel SA mechanism that has linear complexity. The main approach of this work is to eliminate nonlinearity from the original SA. We factorize the matrix multiplication of the SA mechanism without complicated linear approximation. By modifying only a few lines of code from the original SA, the proposed models outperform most transformer-based models on image classification and dense prediction tasks on most capacity regimes.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 48 of its 54 references that are in the map. First posted on arXiv 29 Sep 2021.

Builds on

48 of 54 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2017Mask R-CNNcited by 7,259
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  11. 2016Layer Normalizationcited by 5,579
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
36 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  3. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  4. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  5. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  6. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  7. 2020Reformer: The Efficient Transformercited by 1,658
  8. 2020Rethinking Attention with Performerscited by 1,584
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  10. 2018Group Normalizationcited by 1,514
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  13. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  14. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  15. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  16. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  17. 2016Deep Networks with Stochastic Depthcited by 973
  18. 2020Designing Network Design Spacescited by 954
  19. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  20. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  21. 2021Going deeper with Image Transformerscited by 759
  22. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  23. 2017Large Batch Training of Convolutional Networkscited by 529
  24. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  25. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  26. 2021Early Convolutions Help Transformers See Bettercited by 395
  27. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  28. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  29. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  30. 2019Axial Attention in Multidimensional Transformerscited by 350
  31. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  32. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  33. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 261
  34. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  35. 2019Adaptive Attention Span in Transformerscited by 148
  36. 2021Combining Transformer Generators with Convolutional Discriminatorscited by 5

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 2; 2024: 2; 2025: 1.

11 papers.

  1. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 263
  2. 2022DaViT: Dual Attention Vision Transformerscited by 174
  3. 2023MB-TaylorFormer: Multi-branch Efficient Transformer Expanded by Taylor Formula for Image Dehazingcited by 81
  4. 2022Hydra Attention: Efficient Attention with Many Headscited by 55
  5. 2024Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticketcited by 28
  6. 2022HiViT: Hierarchical Vision Transformer Meets Masked Image Modelingcited by 23
  7. 2022MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attentioncited by 19
  8. 2022FsaNet: Frequency Self-attention for Semantic Segmentationcited by 10
  9. 2023PriViT: Vision Transformers for Fast Private Inferencecited by 4
  10. 2024Transformer-Based Approaches for Sensor-Based Human Activity Recognition: Opportunities and Challengescited by 4
  11. 2025vGamba: Attentive State Space Bottleneck for efficient Long-range Dependencies in Visual Recognitioncited by 0