mlsanityData to 13 Oct 2026

Would you still call this Dax? Novel Visual References in VLMs and Humans

arXiv 2606.05409 ↗First posted 3 Jun 2026cs.CV cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs), like human learners, are frequently exposed to new visual concepts, but how they map novel visual references to language after exposure remains largely underexplored, particularly when those references contradict prior knowledge from pre-training. To study this, we present the Novel Visual References Dataset (NVRD): 19,176 images spanning 90 visual concepts across different levels of visual novelty, each with up to 20 increasingly perturbed versions of the original object to probe generalization. Unlike prior work on visual augmentations of familiar concepts, NVRD comprises entirely novel, open-ended stimuli constructed from scratch, mirroring how humans encounter genuinely new concepts. We evaluate 3 open- and 2 closed-source models alongside 2,400 human judgments for direct human-model comparison, and find that (i) models struggle to acquire novel concepts in-context when they contradict prior knowledge, and (ii) while models and humans show correlated sensitivity to visual perturbations, models significantly overgeneralize, extending learned labels to stimuli that humans reject. We contribute NVRD as a corpus and benchmark for research on visual concept learning in both humans and machines.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 69 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

20 of 69 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models25 Sep
  2. 2024Can We Talk Models Into Seeing the World Differently?14 Mar
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 17

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,191
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,714
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,847
  6. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  7. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 659
  8. 2018Recognition in Terra Incognitacited by 656
  9. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 343
  10. 2018Generalisation in humans and deep neural networkscited by 237
  11. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 178
  12. 2016Understanding How Image Quality Affects Deep Neural Networkscited by 119
5 more
  1. 2023MEWL: Few-shot multimodal word learning with referential uncertaintycited by 17
  2. 2021The Emergence of the Shape Bias Results from Communicative Efficiencycited by 13
  3. 2025Neologism Learning for Controllability and Self-Verbalizationcited by 5
  4. 2025Rapid Word Learning Through Meta In-Context Learningcited by 4
  5. 2025WUGNECTIVES: Novel Entity Inferences of Language Models from Discourse Connectivescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Limits of Learning from Pictures and Text: Vision-Language Models and Embodied Scene Understanding
  2. 2025Visual Language Models show widespread visual deficits on neuropsychological tests
  3. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  4. 2025Do Vision-Language Models See Visualizations Like Humans? Alignment in Chart Categorization
  5. 2024If you can describe it, they can see it: Cross-Modal Learning of Visual Concepts from Textual Descriptions
  6. 2025Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training
  7. 2025The in-context inductive biases of vision-language models differ across modalities
  8. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Models