mlsanityData to 14 Oct 2026

The Curious Case of Visual Grounding: Different Effects for Speech- and Text-based Language Encoders

arXiv 2509.15837 ↗First posted 19 Sep 2025cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

How does visual information included in training affect language processing in audio- and text-based deep learning models? We explore how such visual grounding affects model-internal representations of words, and find substantially different effects in speech- vs. text-based language encoders. Firstly, global representational comparisons reveal that visual grounding increases alignment between representations of spoken and written language, but this effect seems mainly driven by enhanced encoding of word identity rather than meaning. We then apply targeted clustering analyses to probe for phonetic vs. semantic discriminability in model representations. Speech-based representations remain phonetically dominated with visual grounding, but in contrast to text-based representations, visual grounding does not improve semantic discriminability. Our findings could usefully inform the development of more efficient methods to enrich speech-based models with visually-informed semantics.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 18 of its 30 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

18 of 30 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  6. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  7. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 71
  8. 2023What Do Self-Supervised Speech Models Know About Words?cited by 64
  9. 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 51
  10. 2023Cross-Modal Retrieval: A Systematic Review of Methods and Future Directionscited by 40
  11. 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
  12. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
6 more
  1. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  2. 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
  3. 2024Human-like Linguistic Biases in Neural Speech Models: Phonetic Categorization and Phonotactic Constraints in Wav2Vec2.0cited by 19
  4. 2021Explainable Semantic Space by Grounding Language to Vision with Cross-Modal Contrastive Learningcited by 9
  5. 2025What do self-supervised speech models know about Dutch? Analyzing advantages of language-specific pre-trainingcited by 4
  6. 2025From Words to Waves: Analyzing Concept Formation in Speech and Text-Based Foundation Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Tracking the emergence of linguistic structure in self-supervised models learning from speechcited by 1