mlsanityData to 13 Oct 2026

Textless NLP -- Zero Resource Challenge with Low Resource Compute

arXiv 2409.19015 ↗First posted 24 Sep 2024cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

This work addresses the persistent challenges of substantial training time and GPU resource requirements even when training lightweight encoder-vocoder models for Textless NLP. We reduce training steps significantly while improving performance by a) leveraging learning rate schedulers for efficient and faster convergence b) optimizing hop length and c) tuning the interpolation scale factors for better audio quality. Additionally, we explore the latent space representation for Indian languages such as Tamil and Bengali for the acoustic unit discovery and voice conversion task. Our approach leverages a quantized encoder architecture, in conjunction with a vocoder which utilizes the proposed mixture of optimized hop length, tuned interpolation scale factors and a cyclic learning rate scheduler. We obtain consistently good results across English, Tamil and Bengali datasets. The proposed method excels in capturing complex linguistic patterns, resulting in clear reconstructed audio during voice conversion with significantly reduced training time.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 16 of its 19 references that are in the map. First posted on arXiv 24 Sep 2024.

Builds on

16 of 19 references.

References · 16

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,712
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,021
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,870
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,313
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,148
  6. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  7. 2018A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decaycited by 225
  8. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 159
  9. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  10. 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
  11. 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
  12. 2022Deep Fourier Up-Samplingcited by 27
4 more
  1. 2021Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processingcited by 26
  2. 2022Subword Dictionary Learning and Segmentation Techniques for Automatic Speech Recognition in Tamil and Kannadacited by 10
  3. 2021VocBench: A Neural Vocoder Benchmark for Speech Synthesiscited by 8
  4. 2022Knowledge-driven Subword Grammar Modeling for Automatic Speech Recognition in Tamil and Kannadacited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Llama-Mimi: Exploring the Limits of Flattened Speech Language Modeling
  2. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Coding
  3. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis
  4. 2026Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via a Resource-Aware Mixture of Speech Encoders
  5. 2026Voxtral TTS
  6. 2025TESU-LLM: Training Speech-LLMs Without Speech via Unified Encoder Alignment
  7. 2025A2TTS: TTS for Low Resource Indian Languages
  8. 2025Training and Inference Efficiency of Encoder-Decoder Speech Models