mlsanityData to 7 Oct 2026

Towards General Purpose Vision Systems

arXiv 2104.00743 ↗First posted 1 Apr 2021cs.CV cs.AI cs.CLcited by 46

Not on the map. Why?

Abstract (arXiv)

Computer vision systems today are primarily N-purpose systems, designed and trained for a predefined set of tasks. Adapting such systems to new tasks is challenging and often requires non-trivial modifications to the network architecture (e.g. adding new output heads) or training process (e.g. adding new losses). To reduce the time and expertise required to develop new applications, we would like to create general purpose vision systems that can learn and perform a range of tasks without any modification to the architecture or learning process. In this paper, we propose GPV-1, a task-agnostic vision-language architecture that can learn and perform tasks that involve receiving an image and producing text and/or bounding boxes, including classification, localization, visual question answering, captioning, and more. We also propose evaluations of generality of architecture, skill-concept transfer, and learning efficiency that may inform future work on general purpose vision. Our experiments indicate GPV-1 is effective at multiple tasks, reuses some concept knowledge across tasks, can perform the Referring Expressions task zero-shot, and further improves upon the zero-shot performance using a few training samples.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 57 of its 85 references that are in the map. First posted on arXiv 1 Apr 2021.

Builds on

57 of 85 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2017Mask R-CNNcited by 7,259
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  8. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  13. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  14. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  15. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  16. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  17. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  18. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  19. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  20. 2019Green AIcited by 620
  21. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  22. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  23. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  24. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  25. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  26. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  27. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  28. 2016Visual Dialogcited by 377
  29. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  30. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  31. 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
  32. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 197
  33. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  34. 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
  35. 2015Ask Me Anything: Dynamic Memory Networks for Natural Language Processingcited by 116
  36. 2015Grounding of Textual Phrases in Images by Reconstructioncited by 112
  37. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  38. 2018Neural Baby Talkcited by 107

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 11; 2023: 18; 2024: 7; 2025: 7.

46 papers.

Sort:

By area:

  1. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  2. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  3. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  4. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  5. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  6. 2022SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understandingcited by 164
  7. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 162
  8. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  9. 2023Audio-Visual LLM for Video Understandingcited by 76
  10. 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
  11. 2024Affordance-based Robot Manipulation with Flow Matchingcited by 54
  12. 2023LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editingcited by 47
34 more
  1. 2022Webly Supervised Concept Expansion for General Purpose Vision Modelscited by 45
  2. 2022PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Modelscited by 30
  3. 2022GRIT: General Robust Image Task Benchmarkcited by 26
  4. 2023DesCo: Learning Object Recognition with Rich Language Descriptionscited by 25
  5. 2023Type-to-Track: Retrieve Any Object via Prompt-based Trackingcited by 24
  6. 2023DAMEX: Dataset-aware Mixture-of-Experts for visual understanding of mixture-of-datasetscited by 23
  7. 2023Towards Grounded Visual Spatial Reasoning in Multi-Modal Vision Language Modelscited by 18
  8. 2023Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation APIcited by 18
  9. 2023VaQuitA: Enhancing Alignment in LLM-Assisted Video Understandingcited by 17
  10. 2021Hey AI, Can You Solve Complex Tasks by Talking to Agents?cited by 13
  11. 2022FindIt: Generalized Localization with Natural Language Queriescited by 13
  12. 2024Locality Alignment Improves Vision-Language Modelscited by 13
  13. 2023Multitask Multimodal Prompted Training for Interactive Embodied Task Completioncited by 9
  14. 2024Universal Novelty Detection Through Adaptive Contrastive Learningcited by 9
  15. 2023Continual Learning in Open-vocabulary Classification with Complementary Memory Systemscited by 8
  16. 2022A Survey of Computer Vision Technologies In Urban and Controlled-environment Agriculturecited by 6
  17. 2023MINOTAUR: Multi-task Video Grounding From Multimodal Queriescited by 6
  18. 2025Logic-RAG: Augmenting Large Multimodal Models with Visual-Spatial Knowledge for Road Scene Understandingcited by 6
  19. 2022Universal Object Detection with Large Vision Modelcited by 5
  20. 2023Token Boosting for Robust Self-Supervised Visual Transformer Pre-trainingcited by 4
  21. 2025SAGE: A Visual Language Model for Anomaly Detection via Fact Enhancement and Entropy-aware Alignmentcited by 4
  22. 2024Q-GroundCAM: Quantifying Grounding in Vision Language Models via GradCAMcited by 3
  23. 2023Exposing and Addressing Cross-Task Inconsistency in Unified Vision-Language Modelscited by 2
  24. 2024Structured Spatial Reasoning with Open Vocabulary Object Detectorscited by 2
  25. 2025Benchmarking Large and Small MLLMscited by 2
  26. 2022ASC me to Do Anything: Multi-task Training for Embodied AIcited by 1
  27. 2023ViM: Vision Middleware for Unified Downstream Transferringcited by 1
  28. 2024VDebugger: Harnessing Execution Feedback for Debugging Visual Programscited by 1
  29. 2025Cross-Model Transfer of Task Vectors via Few-Shot Orthogonal Alignmentcited by 1
  30. 2025Light as Deception: GPT-driven Natural Relighting Against Vision-Language Pre-training Modelscited by 1
  31. 2023Apollo: Zero-shot MultiModal Reasoning with Multiple Expertscited by 0
  32. 2024Unlocking the Potential of Weakly Labeled Data: A Co-Evolutionary Learning Framework for Abnormality Detection and Report Generationcited by 0
  33. 2025ResNetVLLM -- Multi-modal Vision LLM for the Video Understanding Taskcited by 0
  34. 2025IKIWISI: An Interactive Visual Pattern Generator for Evaluating the Reliability of Vision-Language Models Without Ground Truthcited by 0