mlsanityData to 14 Oct 2026

Cross-Modal Taxonomic Generalization in (Vision-) Language Models

arXiv 2603.07474 ↗First posted 8 Mar 2026cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

What is the interplay between semantic representations learned by language models (LM) from surface form alone to those learned from more grounded evidence? We study this question for a scenario where part of the input comes from a different modality -- in our case, in a vision-language model (VLM), where a pretrained LM is aligned with a pretrained image encoder. As a case study, we focus on the task of predicting hypernyms of objects represented in images. We do so in a VLM setup where the image encoder and LM are kept frozen, and only the intermediate mappings are learned. We progressively deprive the VLM of explicit evidence for hypernyms, and test whether knowledge of hypernyms is recoverable from the LM. We find that the LMs we study can recover this knowledge and generalize even in the most extreme version of this experiment (when the model receives no evidence of a hypernym during training). Additional experiments suggest that this cross-modal taxonomic generalization persists under counterfactual image-label mappings only when the counterfactual data have high visual similarity within each category. Taken together, these findings suggest that cross-modal generalization in LMs arises as a result of both coherence in the extralinguistic input and knowledge derived from language cues.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 26 of its 70 references that are in the map. First posted on arXiv 8 Mar 2026.

Builds on

26 of 70 references.

References · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  7. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  8. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  9. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  10. 2022Linearly Mapping from Image to Text Spacecited by 142
  11. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 126
  12. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 119
14 more
  1. 2025Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areascited by 95
  2. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  3. 2024Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNscited by 65
  4. 2024Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidencecited by 35
  5. 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfactscited by 32
  6. 2021Language Models Use Monotonicity to Assess NPI Licensingcited by 31
  7. 2023Do Vision and Language Models Share Concepts? A Vector Space Alignment Studycited by 27
  8. 2025Can Language Models Learn Typologically Implausible Languages?cited by 23
  9. 2025How Do Vision-Language Models Process Conflicting Information Across Modalities?cited by 23
  10. 2025Both Direct and Indirect Evidence Contribute to Dative Alternation Preferences in Language Modelscited by 18
  11. 2024Characterizing the Role of Similarity in the Property Inferences of Language Modelscited by 11
  12. 2021Explanatory models in neuroscience: Part 2 -- constraint-based intelligibilitycited by 7
  13. 2025Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter Itcited by 6
  14. 2016Relations such as Hypernymy: Identifying and Exploiting Hearst Patterns in Distributional Vectors for Lexical Entailmentcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026(V)LMs generalize beyond surface co-occurrence: Evidence from cross-modal number agreementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors
  2. 2024Vision-Language Models Create Cross-Modal Task Representations
  3. 2024Rethinking VLMs and LLMs for Image Classification
  4. 2026Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recycling
  5. 2022When and why vision-language models behave like bags-of-words, and what to do about it?
  6. 2025Interpreting the linear structure of vision-language model embedding spaces
  7. 2026Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability
  8. 2025How Visual Representations Map to Language Feature Space in Multimodal LLMs