mlsanityData to 7 Oct 2026

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

arXiv 1602.07332 ↗First posted 23 Feb 2016cs.CV cs.AIcited by 2,646

Not on the map. Why?

Cited by 644 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering. Cognition is core to tasks that involve not just recognizing, but reasoning about our visual world. However, models used to tackle the rich content in images for cognitive tasks are still being trained using the same datasets designed for perceptual tasks. To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in an image. When asked "What vehicle is the person riding?", computers will need to identify the objects in an image as well as the relationships riding(man, carriage) and pulling(horse, carriage) in order to answer correctly that "the person is riding a horse-drawn carriage". In this paper, we present the Visual Genome dataset to enable the modeling of such relationships. We collect dense annotations of objects, attributes, and relationships within each image to learn these models. Specifically, our dataset contains over 100K images where each image has an average of 21 objects, 18 attributes, and 18 pairwise relationships between objects. We canonicalize the objects, attributes, relationships, and noun phrases in region descriptions and questions answer pairs to WordNet synsets. Together, these annotations represent the densest and largest dataset of image descriptions, objects, attributes, relationships, and question answers.

Key facts. Cited by 2,646 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Feb 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 406; 2022: 485; 2023: 600; 2024: 540; 2025: 387; 2026: 228.

2,646 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
38 more
  1. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  3. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  4. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  5. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  6. 2021Grounded Language-Image Pre-trainingcited by 1,147
  7. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  8. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  10. 2021Transformers in Vision: A Surveycited by 1,009
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  12. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  13. 2023A Survey on Multimodal Large Language Modelscited by 898
  14. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  15. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  16. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  17. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  18. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  19. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  20. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  21. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  22. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  23. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  24. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  25. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  26. 2024Yi: Open Foundation Models by 01.AIcited by 706
  27. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  28. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  29. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  30. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  31. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  32. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  33. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  34. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  35. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  36. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  37. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  38. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548

50 most cited of 2,646 All 2,646 (JSON)