mlsanityData to 7 Oct 2026

Ripple Attention for Visual Perception with Sub-quadratic Complexity

arXiv 2110.02453 ↗First posted 6 Oct 2021cs.CV cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Transformer architectures are now central to sequence modeling tasks. At its heart is the attention mechanism, which enables effective modeling of long-term dependencies in a sequence. Recently, transformers have been successfully applied in the computer vision domain, where 2D images are first segmented into patches and then treated as 1D sequences. Such linearization, however, impairs the notion of spatial locality in images, which bears important visual clues. To bridge the gap, we propose ripple attention, a sub-quadratic attention mechanism for vision transformers. Built upon the recent kernel-based efficient attention mechanisms, we design a novel dynamic programming algorithm that weights contributions of different tokens to a query with respect to their relative spatial distances in the 2D space in linear observed time. Extensive experiments and analyses demonstrate the effectiveness of ripple attention on various visual tasks.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 56 of its 71 references that are in the map. First posted on arXiv 6 Oct 2021.

Builds on

56 of 71 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
38 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  2. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  3. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  4. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  8. 2020Rethinking Attention with Performerscited by 1,584
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  11. 2017Random Erasing Data Augmentationcited by 1,235
  12. 2018Self-Attention with Relative Position Representationscited by 1,112
  13. 2021Video Swin Transformercited by 1,090
  14. 2016Deep Networks with Stochastic Depthcited by 973
  15. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  16. 2020Pre-Trained Image Processing Transformercited by 891
  17. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  18. 2020Efficient Transformers: A Surveycited by 784
  19. 2018Image Transformercited by 770
  20. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  21. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  22. 2018Universal Transformerscited by 613
  23. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  24. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  25. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  26. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  27. 2021Conditional DETR for Fast Training Convergencecited by 422
  28. 2021Early Convolutions Help Transformers See Bettercited by 395
  29. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  30. 2019Axial Attention in Multidimensional Transformerscited by 350
  31. 2021Random Feature Attentioncited by 350
  32. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  33. 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
  34. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  35. 2021ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Biascited by 216
  36. 2021Fast Convergence of DETR with Spatially Modulated Co-Attentioncited by 176
  37. 2020How Much Position Information Do Convolutional Neural Networks Encode?cited by 171
  38. 2021RegionViT: Regional-to-Local Attention for Vision Transformerscited by 126

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1.

2 papers.

  1. 2022Linear Complexity Randomized Self-attention Mechanismcited by 31
  2. 2023Efficient Attention via Control Variatescited by 23