mlsanityData to 7 Oct 2026

Attention mechanisms and deep learning for machine vision: A survey of the state of the art

arXiv 2106.07550 ↗First posted 3 Jun 2021cs.CV cs.AI cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

With the advent of state of the art nature-inspired pure attention based models i.e. transformers, and their success in natural language processing (NLP), their extension to machine vision (MV) tasks was inevitable and much felt. Subsequently, vision transformers (ViTs) were introduced which are giving quite a challenge to the established deep learning based machine vision techniques. However, pure attention based models/architectures like transformers require huge data, large training times and large computational resources. Some recent works suggest that combinations of these two varied fields can prove to build systems which have the advantages of both these fields. Accordingly, this state of the art survey paper is introduced which hopefully will help readers get useful information about this interesting and potential research area. A gentle introduction to attention mechanisms is given, followed by a discussion of the popular attention based deep architectures. Subsequently, the major categories of the intersection of attention mechanisms and deep learning for machine vision (MV) based are discussed. Afterwards, the major algorithms, issues and trends within the scope of the paper are discussed.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 74 of its 111 references that are in the map. First posted on arXiv 3 Jun 2021.

Builds on

74 of 111 references.

References · 74

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  11. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  12. 2017Squeeze-and-Excitation Networkscited by 4,763
38 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2017The Kinetics Human Action Video Datasetcited by 1,894
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  9. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  11. 2021Transformers in Vision: A Surveycited by 1,009
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  13. 2020Designing Network Design Spacescited by 954
  14. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  15. 2020Pre-Trained Image Processing Transformercited by 891
  16. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  17. 2017Dense-Captioning Events in Videoscited by 789
  18. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  19. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  20. 2015Holistically-Nested Edge Detectioncited by 590
  21. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  22. 2017AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wildcited by 490
  23. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  24. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  25. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  26. 2017Residual Attention Network for Image Classificationcited by 385
  27. 2014Deeply-Supervised Netscited by 364
  28. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  29. 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 346
  30. 2018Video Action Transformer Networkcited by 316
  31. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 308
  32. 2018Few-Shot Learning via Embedding Adaptation with Set-to-Set Functionscited by 293
  33. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  34. 2016SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioningcited by 219
  35. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  36. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  37. 2015Visual Saliency Based on Multiscale Deep Featurescited by 185
  38. 2020CrossTransformers: spatially-aware few-shot transfercited by 174

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 1; 2026: 1.

8 papers.

  1. 2024A Comprehensive Survey on Machine Learning Driven Material Defect Detectioncited by 5
  2. 2024Automated Discovery of Functional Actual Causes in Complex Environmentscited by 3
  3. 2023WATUNet: A Deep Neural Network for Segmentation of Volumetric Sweep Imaging Ultrasoundcited by 1
  4. 2023Enhancing Face Recognition with Latent Space Data Augmentation and Facial Posture Reconstructioncited by 0
  5. 2023Empowering Precision Medicine: AI-Driven Schizophrenia Diagnosis via EEG Signals: A Comprehensive Review from 2002-2023cited by 0
  6. 2024DCNN: Dual Cross-current Neural Networks Realized Using An Interactive Deep Learning Discriminator for Fine-grained Objectscited by 0
  7. 2025A Second-Order Attention Mechanism For Prostate Cancer Segmentation and Detection in Bi-Parametric MRIcited by 0
  8. 2026Gradient-Skipping Relevance Propagation for Efficient Explainability of Vision Transformerscited by 0