# Would you still call this Dax? Novel Visual References in VLMs and Humans

arXiv: https://arxiv.org/abs/2606.05409 · HTML: https://mlsanity.com/paper/2606.05409/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 20 of its 69 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 20 of 20)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models](https://mlsanity.com/paper/2409.17146/) (2024)
- [Generalisation in humans and deep neural networks](https://mlsanity.com/paper/1808.08750/) (2018)
- [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](https://mlsanity.com/paper/2212.07796/) (2022)
- [Understanding How Image Quality Affects Deep Neural Networks](https://mlsanity.com/paper/1604.04004/) (2016)
- [Can We Talk Models Into Seeing the World Differently?](https://mlsanity.com/paper/2403.09193/) (2024)
- [MEWL: Few-shot multimodal word learning with referential uncertainty](https://mlsanity.com/paper/2306.00503/) (2023)
- [The Emergence of the Shape Bias Results from Communicative Efficiency](https://mlsanity.com/paper/2109.06232/) (2021)
- [Neologism Learning for Controllability and Self-Verbalization](https://mlsanity.com/paper/2510.08506/) (2025)
- [Rapid Word Learning Through Meta In-Context Learning](https://mlsanity.com/paper/2502.14791/) (2025)
- [WUGNECTIVES: Novel Entity Inferences of Language Models from Discourse Connectives](https://mlsanity.com/paper/2510.09556/) (2025)

## Abstract (arXiv)

Vision-language models (VLMs), like human learners, are frequently exposed to new visual concepts, but how they map novel visual references to language after exposure remains largely underexplored, particularly when those references contradict prior knowledge from pre-training. To study this, we present the Novel Visual References Dataset (NVRD): 19,176 images spanning 90 visual concepts across different levels of visual novelty, each with up to 20 increasingly perturbed versions of the original object to probe generalization. Unlike prior work on visual augmentations of familiar concepts, NVRD comprises entirely novel, open-ended stimuli constructed from scratch, mirroring how humans encounter genuinely new concepts. We evaluate 3 open- and 2 closed-source models alongside 2,400 human judgments for direct human-model comparison, and find that (i) models struggle to acquire novel concepts in-context when they contradict prior knowledge, and (ii) while models and humans show correlated sensitivity to visual perturbations, models significantly overgeneralize, extending learned labels to stimuli that humans reject. We contribute NVRD as a corpus and benchmark for research on visual concept learning in both humans and machines.