mlsanityData to 7 Oct 2026

Learning Phone Recognition from Unpaired Audio and Phone Sequences Based on Generative Adversarial Network

arXiv 2207.14568 ↗First posted 29 Jul 2022cs.SD cs.CL eess.AScited by 1

Not on the map. Why?

Abstract (arXiv)

ASR has been shown to achieve great performance recently. However, most of them rely on massive paired data, which is not feasible for low-resource languages worldwide. This paper investigates how to learn directly from unpaired phone sequences and speech utterances. We design a two-stage iterative framework. GAN training is adopted in the first stage to find the mapping relationship between unpaired speech and phone sequence. In the second stage, another HMM model is introduced to train from the generator's output, which boosts the performance and provides a better segmentation for the next iteration. In the experiment, we first investigate different choices of model designs. Then we compare the framework to different types of baselines: (i) supervised methods (ii) acoustic unit discovery based methods (iii) methods learning from unpaired data. Our framework performs consistently better than all acoustic unit discovery methods and previous methods learning from unpaired data based on the TIMIT dataset.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 41 of its 64 references that are in the map. First posted on arXiv 29 Jul 2022.

Builds on

41 of 64 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  4. 2017Improved Training of Wasserstein GANscited by 2,223
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  7. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  8. 2017Wasserstein GANcited by 635
  9. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  10. 2017Word Translation Without Parallel Datacited by 480
  11. 2015Attention-Based Models for Speech Recognitioncited by 400
  12. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
29 more
  1. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  2. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  3. 2017Unsupervised Neural Machine Translationcited by 148
  4. 2021Unsupervised Speech Recognitioncited by 147
  5. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 96
  6. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  7. 2016Fully Character-Level Neural Machine Translation without Explicit Segmentationcited by 55
  8. 2018Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speechcited by 52
  9. 2020Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentationcited by 43
  10. 2020The Zero Resource Speech Challenge 2020: Discovering discrete subword and word unitscited by 39
  11. 2019The Zero Resource Speech Challenge 2019: TTS without Tcited by 38
  12. 2016Audio Word2Vec: Unsupervised Learning of Audio Segment Representations using Sequence-to-sequence Autoencodercited by 34
  13. 2018Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spacescited by 34
  14. 2018Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder modelscited by 29
  15. 2015Deep convolutional acoustic word embeddings using word-pair side informationcited by 27
  16. 2016Discriminative Acoustic Word Embeddings: Recurrent Neural Network-Based Approachescited by 25
  17. 2020Towards unsupervised phone and word segmentation using self-supervised vector-quantized neural networkscited by 24
  18. 2017Query-by-Example Search with Discriminative Neural Acoustic Word Embeddingscited by 22
  19. 2018Completely Unsupervised Phoneme Recognition by Adversarially Learning Mapping Relationships from Audio Embeddingscited by 22
  20. 2016Multi-view Recurrent Neural Acoustic Word Embeddingscited by 20
  21. 2018Unsupervised Speech Recognition via Segmental Empirical Output Distribution Matchingcited by 20
  22. 2019Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networkscited by 20
  23. 2017Towards speech-to-text translation without speech recognitioncited by 11
  24. 2016Blind phoneme segmentation with temporal prediction errorscited by 9
  25. 2017Gate Activation Signal Analysis for Gated Recurrent Neural Networks and Its Correlation with Phoneme Boundariescited by 9
  26. 2019Bayesian Subspace Hidden Markov Model for Acoustic Unit Discoverycited by 9
  27. 2020A Hierarchical Subspace Model for Language-Attuned Acoustic Unit Discoverycited by 9
  28. 2020Unsupervised Acoustic Unit Representation Learning for Voice Conversion using WaveNet Auto-encoderscited by 8
  29. 2018Towards Unsupervised Speech-to-Text Translationcited by 6

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Towards Matching Phones and Speech Representationscited by 1