mlsanityData to 14 Oct 2026

Representations of language in a model of visually grounded speech signal

arXiv 1702.01991 ↗First posted 7 Feb 2017cs.CL cs.AI cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

We present a visually grounded model of speech perception which projects spoken utterances and images to a joint semantic space. We use a multi-layer recurrent highway network to model the temporal nature of spoken speech, and show that it learns to extract both form and meaning-based linguistic knowledge from the input signal. We carry out an in-depth analysis of the representations used by different components of the trained model and show that encoding of semantic aspects tends to become richer as we go up the hierarchy of layers, whereas encoding of form-related aspects of the language input tends to initially increase and then plateau or decrease.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). First posted on arXiv 7 Feb 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 16; 2022: 11; 2023: 4; 2024: 3; 2025: 1; 2026: 2.

37 papers.

Sort:

By area:

  1. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  2. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  3. 2021Grounding 'Grounding' in NLPcited by 44
  4. 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
  5. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  6. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  7. 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
  8. 2021Talk, Don't Write: A Study of Direct Speech-Based Image Retrievalcited by 19
  9. 2021Attention-Based Keyword Localisation in Speech using Visual Groundingcited by 13
  10. 2021Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? -- A computational investigationcited by 12
  11. 2021What do End-to-End Speech Models Learn about Speaker, Language and Channel Information? A Layer-wise and Neuron-level Analysiscited by 9
  12. 2021Evaluation of Audio-Visual Alignments in Visually Grounded Speech Modelscited by 8
25 more
  1. 2021Cascaded Multilingual Audio-Visual Learning from Videoscited by 8
  2. 2022Learning English with Peppa Pigcited by 8
  3. 2022YFACC: A Yor`ub'a speech-image dataset for cross-lingual keyword localisation through visual groundingcited by 8
  4. 2021Double Articulation Analyzer with Prosody for Unsupervised Word and Phoneme Discoverycited by 7
  5. 2022Keyword localisation in untranscribed speech using visually grounded speech modelscited by 7
  6. 2021ZR-2021VG: Zero-Resource Speech Challenge, Visually-Grounded Language Modelling track, 2021 editioncited by 6
  7. 2021Unsupervised Automatic Speech Recognition: A Reviewcited by 5
  8. 2022Modeling speech recognition and synthesis simultaneously: Encoding and decoding lexical and sublexical semantic information into speech with no direct access to speech datacited by 5
  9. 2022A Computational Acquisition Model for Multimodal Word Categorizationcited by 5
  10. 2023Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samplescited by 4
  11. 2023Exploring How Generative Adversarial Networks Learn Phonological Representationscited by 4
  12. 2023Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokenscited by 4
  13. 2025Vision-Speech Models: Teaching Speech Models to Converse about Imagescited by 4
  14. 2024Visually Grounded Speech Models have a Mutual Exclusivity Biascited by 3
  15. 2024A model of early word acquisition based on realistic-scale audiovisual naming eventscited by 3
  16. 2021Spoken ObjectNet: A Bias-Controlled Spoken Caption Datasetcited by 2
  17. 2022Seeing the advantage: visually grounding word embeddings to better capture human semantic knowledgecited by 2
  18. 2023Simultaneous or Sequential Training? How Speech Representations Cooperate in a Multi-Task Self-Supervised Learning Systemcited by 2
  19. 2021Voice-assisted Image Labelling for Endoscopic Ultrasound Classification using Neural Networkscited by 1
  20. 2021Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speechcited by 1
  21. 2022Unsupervised Multimodal Word Discovery based on Double Articulation Analysis with Co-occurrence cuescited by 1
  22. 2026Computational modeling of early language learning from acoustic speech and audiovisual input without linguistic priorscited by 1
  23. 2022Modelling word learning and recognition using visually grounded speechcited by 0
  24. 2024Improved Visually Prompted Keyword Localisation in Real Low-Resource Settingscited by 0
  25. 2026Categorize Early, Integrate Late: Divergent Processing Strategies in Automatic Speech Recognitioncited by 0