mlsanityData to 7 Oct 2026

nocaps: novel object captioning at scale

arXiv 1812.08658 ↗First posted 20 Dec 2018cs.CV cs.AI cs.CLcited by 444

Not on the map. Why?

Cited by 213 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Image captioning models have achieved impressive results on datasets containing limited visual concepts and large amounts of paired image-caption training data. However, if these models are to ever function in the wild, a much larger variety of visual concepts must be learned, ideally from less supervision. To encourage the development of image captioning models that can learn visual concepts from alternative data sources, such as object detection datasets, we present the first large-scale benchmark for this task. Dubbed 'nocaps', for novel object captioning at scale, our benchmark consists of 166,100 human-generated captions describing 15,100 images from the OpenImages validation and test sets. The associated training data consists of COCO image-caption pairs, plus OpenImages image-level labels and object bounding boxes. Since OpenImages contains many more classes than COCO, nearly 400 object classes seen in test images have no or very few associated training captions (hence, nocaps). We extend existing novel object captioning models to establish strong baselines for this benchmark and provide analysis to guide future work on this task.

Key facts. Cited by 444 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 49; 2023: 102; 2024: 122; 2025: 93; 2026: 53.

444 papers.

Sort:

By area:

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  11. 2023A Survey on Multimodal Large Language Modelscited by 898
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
38 more
  1. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  2. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  3. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  4. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  5. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  6. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  7. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  8. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  9. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  10. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  11. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  12. 2024What matters when building vision-language models?cited by 321
  13. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  14. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  15. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  16. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  17. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 266
  18. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  19. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  20. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  21. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 240
  22. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  23. 2025ImgEdit: A Unified Image Editing Dataset and Benchmarkcited by 225
  24. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  25. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  26. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  27. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  28. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  29. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  30. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  31. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  32. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  33. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 177
  34. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  35. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  36. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  37. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  38. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161

50 most cited of 444 All 444 (JSON)