From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning
Not on the map. Why?
Abstract (arXiv)
We present a model of visually-grounded language learning based on stacked gated recurrent neural networks which learns to predict visual features given an image description in the form of a sequence of phonemes. The learning task resembles that faced by human language learners who need to discover both structure and meaning from noisy and ambiguous data across modalities. We show that our model indeed learns to predict features of the visual context given phonetically transcribed image descriptions, and show that it represents linguistic information in a hierarchy of levels: lower layers in the stack are comparatively more sensitive to form, whereas higher layers are more sensitive to meaning.
Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Oct 2016.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 4; 2022: 3; 2023: 1.
8 papers.
- 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
- 2022Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoringcited by 24
- 2021Talk, Don't Write: A Study of Direct Speech-Based Image Retrievalcited by 19
- 2021Probing artificial neural networks: insights from neurosciencecited by 17
- 2021Attention-Based Keyword Localisation in Speech using Visual Groundingcited by 13
- 2022Keyword localisation in untranscribed speech using visually grounded speech modelscited by 7
- 2022Towards visually prompted keyword localisation for zero-resource spoken languagescited by 6
- 2023Exploring How Generative Adversarial Networks Learn Phonological Representationscited by 4
- 2023Exploring How Generative Adversarial Networks Learn Phonological Representations21 May 2023
- 2022Towards visually prompted keyword localisation for zero-resource spoken languages12 Oct 2022
- 2022Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoring24 Feb 2022
- 2022Keyword localisation in untranscribed speech using visually grounded speech models2 Feb 2022
- 2021Attention-Based Keyword Localisation in Speech using Visual Grounding16 Jun 2021
- 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques27 Apr 2021
- 2021Probing artificial neural networks: insights from neuroscience16 Apr 2021
- 2021Talk, Don't Write: A Study of Direct Speech-Based Image Retrieval5 Apr 2021