# Cascading and Direct Approaches to Unsupervised Constituency Parsing on Spoken Sentences

arXiv: https://arxiv.org/abs/2303.08809 · HTML: https://mlsanity.com/paper/2303.08809/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 48 references that are in the map.
- First posted on arXiv 15 Mar 2023.

## Builds on (most cited first, 25 of 25)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Constituency Parsing with a Self-Attentive Encoder](https://mlsanity.com/paper/1805.01052/) (2018)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks](https://mlsanity.com/paper/1810.09536/) (2018)
- [The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling](https://mlsanity.com/paper/2011.11588/) (2020)
- [Linguistically-Informed Self-Attention for Semantic Role Labeling](https://mlsanity.com/paper/1804.08199/) (2018)
- [Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoders](https://mlsanity.com/paper/1904.02142/) (2019)
- [Neural Language Modeling by Jointly Learning Syntax and Lexicon](https://mlsanity.com/paper/1711.02013/) (2017)
- [Text-Free Image-to-Speech Synthesis Using Learned Segmental Units](https://mlsanity.com/paper/2012.15454/) (2020)
- [Visually Grounded Neural Syntax Acquisition](https://mlsanity.com/paper/1906.02890/) (2019)
- [Unsupervised Recurrent Neural Network Grammars](https://mlsanity.com/paper/1904.03746/) (2019)
- [The Zero Resource Speech Challenge 2021: Spoken language modelling](https://mlsanity.com/paper/2104.14700/) (2021)
- [Visually Grounded Compound PCFGs](https://mlsanity.com/paper/2009.12404/) (2020)
- [Segmental Contrastive Predictive Coding for Unsupervised Word Segmentation](https://mlsanity.com/paper/2106.02170/) (2021)
- [Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoring](https://mlsanity.com/paper/2202.11929/) (2022)
- [The Return of Lexical Dependencies: Neural Lexicalized PCFGs](https://mlsanity.com/paper/2007.15135/) (2020)
- [Syntactic Structure Distillation Pretraining For Bidirectional Encoders](https://mlsanity.com/paper/2005.13482/) (2020)
- [Neural Bi-Lexicalized PCFG Induction](https://mlsanity.com/paper/2105.15021/) (2021)
- [Dynamic Prosody Generation for Speech Synthesis using Linguistics-Driven Acoustic Embedding Selection](https://mlsanity.com/paper/1912.00955/) (2019)
- [Exploiting Syntactic Features in a Parsed Tree to Improve End-to-End TTS](https://mlsanity.com/paper/1904.04764/) (2019)
- [Unsupervised Word Segmentation using K Nearest Neighbors](https://mlsanity.com/paper/2204.13094/) (2022)
- [Syntactic representation learning for neural network based TTS with syntactic parse tree traversal](https://mlsanity.com/paper/2012.06971/) (2020)
- [Assessing the Use of Prosody in Constituency Parsing of Imperfect Transcripts](https://mlsanity.com/paper/2106.07794/) (2021)

## Built on by (most cited first, 2 of 2)

- [Audio-Visual Neural Syntax Acquisition](https://mlsanity.com/paper/2310.07654/) (2023)
- [Structured Tree Alignment for Evaluation of (Speech) Constituency Parsing](https://mlsanity.com/paper/2402.13433/) (2024)

## Abstract (arXiv)

Past work on unsupervised parsing is constrained to written form. In this paper, we present the first study on unsupervised spoken constituency parsing given unlabeled spoken sentences and unpaired textual data. The goal is to determine the spoken sentences' hierarchical syntactic structure in the form of constituency parse trees, such that each node is a span of audio that corresponds to a constituent. We compare two approaches: (1) cascading an unsupervised automatic speech recognition (ASR) model and an unsupervised parser to obtain parse trees on ASR transcripts, and (2) direct training an unsupervised parser on continuous word-level speech representations. This is done by first splitting utterances into sequences of word-level segments, and aggregating self-supervised speech representations within segments to obtain segment embeddings. We find that separately training a parser on the unpaired text and directly applying it on ASR transcripts for inference produces better results for unsupervised parsing. Additionally, our results suggest that accurate segmentation alone may be sufficient to parse spoken sentences accurately. Finally, we show the direct approach may learn head-directionality correctly for both head-initial and head-final languages without any explicit inductive bias.