Audio-Visual Neural Syntax Acquisition
Not on the map. Why?
Abstract (arXiv)
We study phrase structure induction from visually-grounded speech. The core idea is to first segment the speech waveform into sequences of word segments, and subsequently induce phrase structure using the inferred segment-level continuous representations. We present the Audio-Visual Neural Syntax Learner (AV-NSL) that learns phrase structure by listening to audio and looking at images, without ever being exposed to text. By training on paired images and spoken captions, AV-NSL exhibits the capability to infer meaningful phrase structures that are comparable to those derived by naturally-supervised text parsers, for both English and German. Our findings extend prior work in unsupervised language acquisition from speech and grounded grammar induction, and present one approach to bridge the gap between the two topics.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 29 of its 50 references that are in the map. First posted on arXiv 11 Oct 2023.
Builds on
29 of 50 references.
References · 29
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2016Gaussian Error Linear Units (GELUs)cited by 3,431
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
- 2018Constituency Parsing with a Self-Attentive Encodercited by 196
- 2019Compound Probabilistic Context-Free Grammars for Grammar Inductioncited by 76
- 2016Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learnercited by 68
- 2019Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoderscited by 60
- 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 50
- 2019Visually Grounded Neural Syntax Acquisitioncited by 48
- 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
- 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
17 more
- 2020Visually Grounded Compound PCFGscited by 34
- 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
- 2019rVAD: An Unsupervised Segment-Based Robust Voice Activity Detection Methodcited by 32
- 2021Segmental Contrastive Predictive Coding for Unsupervised Word Segmentationcited by 32
- 2016A segmental framework for fully-unsupervised large-vocabulary speech recognitioncited by 27
- 2017Learning Word-Like Units from Joint Audio-Visual Analysiscited by 27
- 2022Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoringcited by 24
- 2020On the Role of Supervision in Unsupervised Constituency Parsingcited by 18
- 2021Video-aided Unsupervised Grammar Inductioncited by 17
- 2021Aligned Contrastive Predictive Codingcited by 15
- 2019Dynamic Prosody Generation for Speech Synthesis using Linguistics-Driven Acoustic Embedding Selectioncited by 11
- 2019Exploiting Syntactic Features in a Parsed Tree to Improve End-to-End TTScited by 10
- 2019Neural Constituency Parsing of Speech Transcriptscited by 10
- 2020On the Role of Style in Parsing Speech with Neural Modelscited by 7
- 2018An Empirical Analysis of the Correlation of Syntax and Prosodycited by 4
- 2021Assessing the Use of Prosody in Constituency Parsing of Imperfect Transcriptscited by 3
- 2023Cascading and Direct Approaches to Unsupervised Constituency Parsing on Spoken Sentencescited by 2
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 1.
2 papers.