mlsanityData to 14 Oct 2026

Discrete representations in neural models of spoken language

arXiv 2105.05582 ↗First posted 12 May 2021cs.CL cs.LG cs.SDcited by 3

Not on the map. Why?

Abstract (arXiv)

The distributed and continuous representations used by neural networks are at odds with representations employed in linguistics, which are typically symbolic. Vector quantization has been proposed as a way to induce discrete neural representations that are closer in nature to their linguistic counterparts. However, it is not clear which metrics are the best-suited to analyze such discrete representations. We compare the merits of four commonly used metrics in the context of weakly supervised models of spoken language. We compare the results they show when applied to two different models, while systematically studying the effect of the placement and size of the discretization layer. We find that different evaluation regimes can give inconsistent results. While we can attribute them to the properties of the different metrics in most cases, one point of concern remains: the use of minimal pairs of phoneme triples as stimuli disadvantages larger discrete unit inventories, unlike metrics applied to complete utterances. Furthermore, while in general vector quantization induces representations that correlate with units posited in linguistics, the strength of this correlation is only moderate.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 12 May 2021.

Builds on

32 of 44 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  5. 2015Cyclical Learning Rates for Training Neural Networkscited by 629
  6. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  7. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  8. 2020Information-Theoretic Probing for Linguistic Structurecited by 207
  9. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  10. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  11. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  12. 2018From Audio to Semantics: Approaches to end-to-end spoken language understandingcited by 63
20 more
  1. 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
  2. 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
  3. 2017The Zero Resource Speech Challenge 2017cited by 45
  4. 2018How agents see things: On visual representations in an emergent language gamecited by 45
  5. 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
  6. 2020The Zero Resource Speech Challenge 2020: Discovering discrete subword and word unitscited by 39
  7. 2019The Zero Resource Speech Challenge 2019: TTS without Tcited by 38
  8. 2019Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brainscited by 38
  9. 2019Correlating neural and symbolic representations of languagecited by 33
  10. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  11. 2017Encoding of phonology in a recurrent neural model of grounded speechcited by 27
  12. 2019VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019cited by 25
  13. 2019Language learning using Speech to Image retrievalcited by 21
  14. 2020Transformer VQ-VAE for Unsupervised Unit Discovery and Speech Synthesis: ZeroSpeech 2020 Challengecited by 21
  15. 2020Evaluating the reliability of acoustic speech embeddingscited by 21
  16. 2019Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networkscited by 20
  17. 2020Discourse structure interacts with reference but not syntax in neural language modelscited by 17
  18. 2020Analyzing analytical methods: The case of phonology in neural models of spoken languagecited by 16
  19. 2019Higher-order Comparisons of Sentence Encoder Representationscited by 10
  20. 2020Textual Supervision for Visually Grounded Spoken Language Understandingcited by 9

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1.

3 papers.

  1. 2022textless-lib: a Library for Textless Spoken Language Processingcited by 28
  2. 2023An Information-Theoretic Analysis of Self-supervised Discrete Representations of Speechcited by 15
  3. 2022Evaluating context-invariance in unsupervised speech representationscited by 11

Similar, not cited

Close in wording, no citation either way

  1. 2026Lexical Tone is Hard to Quantize: Probing Discrete Speech Units in Mandarin and Yor`ub'a
  2. 2024Estimating the Completeness of Discrete Speech Units
  3. 2024Investigating Disentanglement in a Phoneme-level Speech Codec for Prosody Modeling
  4. 2022Autoregressive Co-Training for Learning Discrete Speech Representations
  5. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey
  6. 2026Early Quantization Shrinks Codebook: A Simple Fix for Diversity-Preserving Tokenization
  7. 2022Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling
  8. 2025Towards Atoms of Large Language Models