mlsanityData to 7 Oct 2026

Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models

arXiv 1505.04870 ↗First posted 19 May 2015cs.CV cs.CLcited by 1,352

Not on the map. Why?

Cited by 439 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The Flickr30k dataset has become a standard benchmark for sentence-based image description. This paper presents Flickr30k Entities, which augments the 158k captions from Flickr30k with 244k coreference chains, linking mentions of the same entities across different captions for the same image, and associating them with 276k manually annotated bounding boxes. Such annotations are essential for continued progress in automatic image description and grounded language understanding. They enable us to define a new benchmark for localization of textual entity mentions in an image. We present a strong baseline for this task that combines an image-text embedding, detectors for common objects, a color classifier, and a bias towards selecting larger objects. While our baseline rivals in accuracy more complex state-of-the-art models, we show that its gains cannot be easily parlayed into improvements on such tasks as image-sentence retrieval, thus underlining the limitations of current methods and the need for further research.

Key facts. Cited by 1,352 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 May 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 110; 2022: 176; 2023: 301; 2024: 289; 2025: 275; 2026: 200.

1,352 papers.

Sort:

By area:

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  8. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  9. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  10. 2021Grounded Language-Image Pre-trainingcited by 1,147
  11. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  12. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
38 more
  1. 2021Transformers in Vision: A Surveycited by 1,009
  2. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  3. 2023A Survey on Multimodal Large Language Modelscited by 898
  4. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  5. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  6. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  7. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  8. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  9. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  10. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  11. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  12. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  13. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  14. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  15. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  16. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  17. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  18. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 498
  19. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  20. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  21. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  22. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  23. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  24. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  25. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  26. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  27. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  28. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  29. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  30. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  31. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  32. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  33. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  34. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  35. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  36. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  37. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  38. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275

50 most cited of 1,352 All 1,352 (JSON)