# A Convolutional Deep Markov Model for Unsupervised Speech Representation Learning

arXiv: https://arxiv.org/abs/2006.02547 · HTML: https://mlsanity.com/paper/2006.02547/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2020.

## Built on by (most cited first, 18 of 18)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation](https://mlsanity.com/paper/2205.08180/) (2022)
- [Textless Speech Emotion Conversion using Discrete and Decomposed Representations](https://mlsanity.com/paper/2111.07402/) (2021)
- [Variable-rate discrete representation learning](https://mlsanity.com/paper/2103.06089/) (2021)
- [Aligned Contrastive Predictive Coding](https://mlsanity.com/paper/2104.11946/) (2021)
- [A Brief Overview of Unsupervised Neural Speech Representation Learning](https://mlsanity.com/paper/2203.01829/) (2022)
- [Pretext Tasks selection for multitask self-supervised speech representation learning](https://mlsanity.com/paper/2107.00594/) (2021)
- [USTED: Improving ASR with a Unified Speech and Text Encoder-Decoder](https://mlsanity.com/paper/2202.06045/) (2022)
- [Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing](https://mlsanity.com/paper/2211.01522/) (2022)
- [On Scaling Contrastive Representations for Low-Resource Speech Recognition](https://mlsanity.com/paper/2102.00850/) (2021)
- [Conditional independence for pretext task selection in Self-supervised speech representation learning](https://mlsanity.com/paper/2104.07388/) (2021)
- [Improved Cross-Lingual Transfer Learning For Automatic Speech Translation](https://mlsanity.com/paper/2306.00789/) (2023)
- [Neural calibration of hidden inhomogeneous Markov chains -- Information decompression in life insurance](https://mlsanity.com/paper/2201.02397/) (2022)
- [Learning Invariant Representation and Risk Minimized for Unsupervised Accent Domain Adaptation](https://mlsanity.com/paper/2210.08182/) (2022)
- [End-to-End Open Vocabulary Keyword Search With Multilingual Neural Representations](https://mlsanity.com/paper/2308.08027/) (2023)
- [Written Term Detection Improves Spoken Term Detection](https://mlsanity.com/paper/2407.04601/) (2024)

## Abstract (arXiv)

Probabilistic Latent Variable Models (LVMs) provide an alternative to self-supervised learning approaches for linguistic representation learning from speech. LVMs admit an intuitive probabilistic interpretation where the latent structure shapes the information extracted from the signal. Even though LVMs have recently seen a renewed interest due to the introduction of Variational Autoencoders (VAEs), their use for speech representation learning remains largely unexplored. In this work, we propose Convolutional Deep Markov Model (ConvDMM), a Gaussian state-space model with non-linear emission and transition functions modelled by deep neural networks. This unsupervised model is trained using black box variational inference. A deep convolutional neural network is used as an inference network for structured variational approximation. When trained on a large scale speech dataset (LibriSpeech), ConvDMM produces features that significantly outperform multiple self-supervised feature extracting methods on linear phone classification and recognition on the Wall Street Journal dataset. Furthermore, we found that ConvDMM complements self-supervised methods like Wav2Vec and PASE, improving on the results achieved with any of the methods alone. Lastly, we find that ConvDMM features enable learning better phone recognizers than any other features in an extreme low-resource regime with few labeled training examples.