mlsanityData to 14 Oct 2026

World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models

arXiv 2306.08685 ↗First posted 14 Jun 2023cs.CL cs.AI cs.CVcited by 10

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The ability to connect language units to their referents in the physical world, referred to as grounding, is crucial to learning and understanding grounded meanings of words. While humans demonstrate fast mapping in new word learning, it remains unclear whether modern vision-language models can truly represent language with their grounded meanings and how grounding may further bootstrap new word learning. To this end, we introduce Grounded Open Vocabulary Acquisition (GOVA) to examine grounding and bootstrapping in open-world language learning. As an initial attempt, we propose object-oriented BERT (OctoBERT), a novel visually-grounded language model by pre-training on image-text pairs highlighting grounding as an objective. Through extensive experiments and analysis, we demonstrate that OctoBERT is a more coherent and fast grounded word learner, and that the grounding ability acquired during pre-training helps the model to learn unseen words more rapidly and robustly. Our code is available at https://github.com/sled-group/world-to-words

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 86 references that are in the map. First posted on arXiv 14 Jun 2023.

Builds on

45 of 86 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone15 Jun
  2. 2022A Survey of Vision-Language Pre-Trained Models18 Feb

Other references · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2020End-to-End Object Detection with Transformerscited by 7,414
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2019Language Models as Knowledge Bases?cited by 1,936
  6. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,368
  7. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  10. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  11. 2021Grounded Language-Image Pre-trainingcited by 1,156
  12. 2016Modeling Context in Referring Expressionscited by 1,063
31 more
  1. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  2. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  3. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  4. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  5. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  6. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
  7. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  8. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  9. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 372
  10. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 360
  11. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  12. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  13. 2020Experience Grounds Languagecited by 242
  14. 2013Zero-Shot Learning Through Cross-Modal Transfercited by 237
  15. 2018Zero-Shot Object Detectioncited by 189
  16. 2015Natural Language Object Retrievalcited by 124
  17. 2020PhraseCut: Language-based Image Segmentation in the Wildcited by 116
  18. 2016Learning Visual N-Grams from Web Datacited by 95
  19. 2020Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervisioncited by 94
  20. 2020Grounded Situation Recognitioncited by 78
  21. 2019Dont Even Look Once: Synthesizing Features for Zero-Shot Detectioncited by 58
  22. 2021Word Acquisition in Neural Language Modelscited by 41
  23. 2015Combining Language and Vision with a Multimodal Skip-gram Modelcited by 28
  24. 2018Zero-Shot Detectioncited by 27
  25. 2022Imagination-Augmented Natural Language Understandingcited by 19
  26. 2022Visually-Augmented Language Modelingcited by 18
  27. 2017R-FCN-3000 at 30fps: Decoupling Detection and Classificationcited by 15
  28. 2015Learning language through picturescited by 12
  29. 2021FIBER: Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Frameworkcited by 12
  30. 2022Grammar-Based Grounded Lexicon Learningcited by 11
  31. 2014Context-based Word Acquisition for Situated Dialogue in a Virtual Worldcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 5; 2026: 1.

10 papers.

  1. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  2. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguitiescited by 59
  3. 2025MindJourney: Test-Time Scaling with World Models for Spatial Reasoningcited by 50
  4. 2025Towards Visual Text Grounding of Multimodal Large Language Modelcited by 18
  5. 2023Object Recognition as Next Token Predictioncited by 12
  6. 2024Babysit A Language Model From Scratch: Interactive Language Learning by Trials and Demonstrationscited by 12
  7. 2025Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generationcited by 8
  8. 2025The Mechanistic Emergence of Symbol Grounding in Language Modelscited by 6
  9. 2025Computer Vision Models Show Human-Like Sensitivity to Geometric and Topological Conceptscited by 2
  10. 2026When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Vision-Language Grounding as Bidirectional Concept Correspondence
  2. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation
  3. 2023Expand BERT Representation with Visual Information via Grounded Language Learning with Multimodal Partial Alignment
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World
  5. 2024Toward Visual Grounding: A Survey
  6. 2023Learning the meanings of function words from grounded language using a visual question answering model
  7. 2023Describe me an Aucklet: Generating Grounded Perceptual Category Descriptions
  8. 2024Learning Visual Grounding from Generative Vision and Language Model