# Unsupervised Multimodal Word Discovery based on Double Articulation Analysis with Co-occurrence cues

arXiv: https://arxiv.org/abs/2201.06786 · HTML: https://mlsanity.com/paper/2201.06786/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 70 references that are in the map.
- First posted on arXiv 18 Jan 2022.

## Builds on (most cited first, 14 of 14)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Unsupervised speech representation learning using WaveNet autoencoders](https://mlsanity.com/paper/1901.08810/) (2019)
- [Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge](https://mlsanity.com/paper/2005.09409/) (2020)
- [Representations of language in a model of visually grounded speech signal](https://mlsanity.com/paper/1702.01991/) (2017)
- [Learning Word-Like Units from Joint Audio-Visual Analysis](https://mlsanity.com/paper/1701.07481/) (2017)
- [Neuro-SERKET: Development of Integrative Cognitive System through the Composition of Deep Probabilistic Generative Models](https://mlsanity.com/paper/1910.08918/) (2019)
- [SERKET: An Architecture for Connecting Stochastic Models to Realize a Large-Scale Cognitive Model](https://mlsanity.com/paper/1712.00929/) (2017)
- [Unsupervised word segmentation and lexicon discovery using acoustic word embeddings](https://mlsanity.com/paper/1603.02845/) (2016)
- [Double Articulation Analyzer with Prosody for Unsupervised Word and Phoneme Discovery](https://mlsanity.com/paper/2103.08199/) (2021)
- [Nonparametric Bayesian Double Articulation Analyzer for Direct Language Acquisition from Continuous Speech Signals](https://mlsanity.com/paper/1506.06646/) (2015)
- [Unsupervised Phoneme and Word Discovery from Multiple Speakers using Double Articulation Analyzer and Neural Network with Parametric Bias](https://mlsanity.com/paper/1906.11049/) (2019)
- [Unsupervised Lexical Acquisition of Relative Spatial Concepts Using Spoken User Utterances](https://mlsanity.com/paper/2106.08574/) (2021)

## Built on by (most cited first, 1 of 1)

- [Brain-inspired probabilistic generative model for double articulation analysis of spoken language](https://mlsanity.com/paper/2207.02457/) (2022)

## Abstract (arXiv)

Human infants acquire their verbal lexicon with minimal prior knowledge of language based on the statistical properties of phonological distributions and the co-occurrence of other sensory stimuli. This study proposes a novel fully unsupervised learning method for discovering speech units using phonological information as a distributional cue and object information as a co-occurrence cue. The proposed method can acquire words and phonemes from speech signals using unsupervised learning and utilize object information based on multiple modalities-vision, tactile, and auditory-simultaneously. The proposed method is based on the nonparametric Bayesian double articulation analyzer (NPB-DAA) discovering phonemes and words from phonological features, and multimodal latent Dirichlet allocation (MLDA) categorizing multimodal information obtained from objects. In an experiment, the proposed method showed higher word discovery performance than baseline methods. Words that expressed the characteristics of objects (i.e., words corresponding to nouns and adjectives) were segmented accurately. Furthermore, we examined how learning performance is affected by differences in the importance of linguistic information. Increasing the weight of the word modality further improved performance relative to that of the fixed condition.