mlsanityData to 7 Oct 2026

Perspectives and Prospects on Transformer Architecture for Cross-Modal Tasks with Language and Vision

arXiv 2103.04037 ↗First posted 6 Mar 2021cs.CV cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Transformer architectures have brought about fundamental changes to computational linguistic field, which had been dominated by recurrent neural networks for many years. Its success also implies drastic changes in cross-modal tasks with language and vision, and many researchers have already tackled the issue. In this paper, we review some of the most critical milestones in the field, as well as overall trends on how transformer architecture has been incorporated into visuolinguistic cross-modal tasks. Furthermore, we discuss its current limitations and speculate upon some of the prospects that we find imminent.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 104 of its 160 references that are in the map. First posted on arXiv 6 Mar 2021.

Builds on

104 of 160 references.

References · 104

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  10. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  11. 2020End-to-End Object Detection with Transformerscited by 7,384
  12. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
38 more
  1. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  2. 2016Layer Normalizationcited by 5,579
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  7. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2015Fast R-CNNcited by 2,930
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  13. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  14. 2015VQA: Visual Question Answeringcited by 2,489
  15. 2018Deep contextualized word representationscited by 2,285
  16. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  17. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  18. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  19. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  20. 2020Reformer: The Efficient Transformercited by 1,658
  21. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  22. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  23. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  24. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  25. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  26. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  27. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  28. 2016Modeling Context in Referring Expressionscited by 1,054
  29. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,018
  30. 2021Transformers in Vision: A Surveycited by 1,009
  31. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  32. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  33. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  34. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  35. 2020Pre-Trained Image Processing Transformercited by 891
  36. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  37. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  38. 2018Image Transformercited by 770

50 most cited of 104 All 104 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 2; 2023: 6; 2024: 2; 2025: 2.

14 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 70
  2. 2022Video Transformers: A Surveycited by 63
  3. 2021Core Challenges in Embodied Vision-Language Planningcited by 30
  4. 2024M4oE: A Foundation Model for Medical Multimodal Image Segmentation with Mixture of Expertscited by 26
  5. 2021Data Efficient Masked Language Modeling for Vision and Languagecited by 18
  6. 2023Distilling Efficient Vision Transformers from CNNs for Semantic Segmentationcited by 18
  7. 2023Transformer-based Generative Adversarial Networks in Computer Vision: A Comprehensive Surveycited by 10
  8. 2025MolSpectra: Pre-training 3D Molecular Representation with Multi-modal Energy Spectracited by 8
  9. 2023Semantic Contrastive Bootstrapping for Single-positive Multi-label Recognitioncited by 5
  10. 2022A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrievalcited by 4
  11. 2023ExpPoint-MAE: Better interpretability and performance for self-supervised point cloud transformerscited by 3
  12. 2024Advances in Transformers for Robotic Applications: A Reviewcited by 2
2 more
  1. 2023Core Challenges in Embodied Vision-Language Planningcited by 0
  2. 2025DIMT25@ICDAR2025: HW-TSC's End-to-End Document Image Machine Translation System Leveraging Large Vision-Language Modelcited by 0