mlsanityData to 7 Oct 2026

Vision-Language Intelligence: Tasks, Representation Learning, and Large Models

arXiv 2203.01922 ↗First posted 3 Mar 2022cs.CV cs.AI cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

This paper presents a comprehensive survey of vision-language (VL) intelligence from the perspective of time. This survey is inspired by the remarkable progress in both computer vision and natural language processing, and recent trends shifting from single modality processing to multiple modality comprehension. We summarize the development in this field into three time periods, namely task-specific methods, vision-language pre-training (VLP) methods, and larger models empowered by large-scale weakly-labeled data. We first take some common VL tasks as examples to introduce the development of task-specific methods. Then we focus on VLP methods and comprehensively review key components of the model structures and training methods. After that, we show how recent work utilizes large-scale raw image-text data to learn language-aligned visual representations that generalize better on zero or few shot learning tasks. Finally, we discuss some potential future trends towards modality cooperation, unified representation, and knowledge incorporation. We believe that this review will be of help for researchers and practitioners of AI and ML, especially those interested in computer vision and natural language processing.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 122 of its 167 references that are in the map. First posted on arXiv 3 Mar 2022.

Builds on

122 of 167 references.

References · 122

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2014Going Deeper with Convolutionscited by 4,231
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  3. 2016Improved Techniques for Training GANscited by 2,938
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  12. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  13. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  14. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  15. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  16. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  17. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  18. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  19. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  20. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  21. 2016Modeling Context in Referring Expressionscited by 1,054
  22. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  23. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  24. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  25. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  26. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  27. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  28. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  29. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  30. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  31. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  32. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  33. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  34. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  35. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  36. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  37. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  38. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451

50 most cited of 122 All 122 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 12; 2024: 9; 2025: 4.

28 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2022Multimodal Learning with Transformers: A Surveycited by 399
  3. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  4. 2025Thyme: Think Beyond Imagescited by 160
  5. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  6. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  7. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  8. 2023Understanding Multimodal Contrastive Learning and Incorporating Unpaired Datacited by 43
  9. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  10. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
  11. 2023UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Modelscited by 23
  12. 2022IDEA: Increasing Text Diversity via Online Multi-Label Recognition for Vision-Language Pre-trainingcited by 14
16 more
  1. 2023Direction-Oriented Visual-semantic Embedding Model for Remote Sensing Image-text Retrievalcited by 14
  2. 2023Robotic Applications of Pre-Trained Vision-Language Models to Various Recognition Behaviorscited by 12
  3. 2024Gaussian Splatting Under Attack: Investigating Adversarial Noise in 3D Objectscited by 11
  4. 2025Self-Rewarding Large Vision-Language Models for Optimizing Prompts in Text-to-Image Generationcited by 10
  5. 2024One missing piece in Vision and Language: A Survey on Comics Understandingcited by 8
  6. 2023ROSGPT_Vision: Commanding Robots Using Only Language Models' Promptscited by 4
  7. 2024I Know About "Up"! Enhancing Spatial Reasoning in Visual Language Models Through 3D Reconstructioncited by 4
  8. 2024Continuous Object State Recognition for Cooking Robots Using Pre-Trained Vision-Language Models and Black-box Optimizationcited by 3
  9. 2023Scene Graph Based Fusion Network For Image-Text Retrievalcited by 2
  10. 2023ELIP: Efficient Discriminative Language-Image Pre-training with Fewer Vision Tokenscited by 2
  11. 2023Daily Assistive View Control Learning of Low-Cost Low-Rigidity Robot via Large-Scale Vision-Language Modelcited by 1
  12. 2024Reflex-Based Open-Vocabulary Navigation without Prior Knowledge Using Omnidirectional Camera and Multiple Vision-Language Modelscited by 1
  13. 2023Binary State Recognition by Robots using Visual Question Answering of Pre-Trained Vision-Language Modelcited by 0
  14. 2024Robotic Environmental State Recognition with Pre-Trained Vision-Language Models and Black-Box Optimizationcited by 0
  15. 2024Robotic State Recognition with Image-to-Text Retrieval Task of Pre-Trained Vision-Language Model and Black-Box Optimizationcited by 0
  16. 2025WildFireCan-MMD: A Multimodal Dataset for Classification of User-Generated Content During Wildfires in Canadacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  2. 2022VLP: A Survey on Vision-Language Pre-training
  3. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Survey
  4. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models
  5. 2025Vision Language Models: A Survey of 26K Papers
  6. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  7. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  8. 2025A Survey on Efficient Vision-Language Models