mlsanityData to 14 Oct 2026

MELD: Mel-Spectrogram-Based Speech Language Modeling with Discrete Latent Variables

arXiv 2605.29859 ↗First posted 28 May 2026eess.AS cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent speech language models rely on encoders that are optimized separately from autoregressive models. Since these encoders are unaware of the downstream objectives, the extracted representations may not be optimal for downstream tasks. To address this limitation, we introduce a discrete latent variable model on mel spectrograms that jointly optimizes the encoder and the speech language model. Joint optimization not only brings improvements over codec-based and other mel-spectrogram-based baselines on zero-shot Text-to-Speech (TTS) and Speech-to-Text (STT) tasks, but also effectively alleviates common issues in autoregressive mel spectrogram modeling, such as prolonged silence generation and word omissions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 43 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

36 of 43 references.

References · 36

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2013Auto-Encoding Variational Bayescited by 4,053
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  7. 2019An Introduction to Variational Autoencoderscited by 1,000
  8. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  9. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  10. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  11. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  12. 2022High Fidelity Neural Audio Compressioncited by 687
24 more
  1. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
  2. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  3. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  4. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  5. 2023Simple and Controllable Music Generationcited by 488
  6. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  7. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  8. 2017Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representationscited by 152
  9. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  10. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  11. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  12. 2023Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networkscited by 82
  13. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  14. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  15. 2024Multimodal Latent Language Modeling with Next-Token Diffusioncited by 48
  16. 2025VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloningcited by 42
  17. 2023A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speechcited by 22
  18. 2024dMel: Speech Tokenization made Simplecited by 11
  19. 2024Estimating the Completeness of Discrete Speech Unitscited by 9
  20. 2024Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representationscited by 8
  21. 2025Flow-SLM: Joint Learning of Linguistic and Acoustic Information for Spoken Language Modelingcited by 7
  22. 2024Robust and Unbounded Length Generalization in Autoregressive Transformer-Based Text-to-Speechcited by 4
  23. 2025Differentiable K-means for Fully-optimized Discrete Token-based ASRcited by 3
  24. 2024Enabling Beam Search for Language Model-Based Text-to-Speech Synthesiscited by 2

Built on by

Not cited yet.