Data to 7 Oct 2026

Connecting Vision and Language with Localized Narratives

arXiv 1912.03098 ↗First posted 6 Dec 2019cs.CVcited by 225

Not on the map. Why?

Cited by 53 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose Localized Narratives, a new form of multimodal image annotations connecting vision and language. We ask annotators to describe an image with their voice while simultaneously hovering their mouse over the region they are describing. Since the voice and the mouse pointer are synchronized, we can localize every single word in the description. This dense visual grounding takes the form of a mouse trace segment per word and is unique to our data. We annotated 849k images with Localized Narratives: the whole COCO, Flickr30k, and ADE20K datasets, and 671k images of Open Images, all of which we make publicly available. We provide an extensive analysis of these annotations showing they are diverse, accurate, and efficient to produce. We also demonstrate their utility on the application of controlled image captioning.

Key facts. Cited by 225 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 33; 2023: 59; 2024: 44; 2025: 43; 2026: 18.

225 papers.

Sort:

By area:

2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
2023Vision-Language Models for Vision Tasks: A Surveycited by 834
2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
2024What matters when building vision-language models?cited by 321
2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321

38 more

2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 266
2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
2024Building and better understanding vision-language models: insights and future directionscited by 169
2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 135
2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
2021ArtEmis: Affective Language for Visual Artcited by 103
2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100
2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 98
2023SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentationcited by 95
2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91
2024LLARVA: Vision-Action Instruction Tuning Enhances Robot Learningcited by 87
2023ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptionscited by 86
2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
2023Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videoscited by 84
2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
2024Compositional Entailment Learning for Hyperbolic Vision-Language Modelscited by 72
2023ALIP: Adaptive Language-Image Pre-training with Synthetic Captioncited by 70
2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
2021Decoupling the Role of Data, Attention, and Losses in Multimodal Transformerscited by 63
2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 62

50 most cited of 225 All 225 (JSON)