mlsanityData to 14 Oct 2026

Taming Audio VAEs via Target-KL Regularization

arXiv 2605.17085 ↗First posted 16 May 2026cs.SD cs.LG eess.AScited by 0

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Latent diffusion models have emerged as the dominant paradigm for many generation tasks including audio generation such as text-to-audio, text-to-music and text-to-speech. A key component of latent diffusion is an autoencoder (VAE) that compresses high-dimensional signals into a low frame rate continuous representation that is conducive for downstream prediction. Regularizing these VAEs is challenging, as there is a trade-off between over-regularized (poor output quality) and under-regularized (difficult to predict) latent representations. We propose a framework for studying this trade-off through compression and train Audio VAEs at specific bitrates via target-KL regularization. This allows direct comparison to well-studied discrete neural audio codec models, and the construction of rate-distortion curves for audio VAEs. We evaluate the impact of target-KL regularization on text-to-sound generation and find that sweeping compression rates is helpful in identifying the optimal generation setting.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 34 of its 40 references that are in the map. First posted on arXiv 16 May 2026.

Builds on

34 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesis8 Jun

Other references · 33

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2022Scalable Diffusion Models with Transformerscited by 6,054
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2019An Introduction to Variational Autoencoderscited by 1,000
  9. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  10. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  11. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
  12. 2022High Fidelity Neural Audio Compressioncited by 687
21 more
  1. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 640
  2. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  3. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  4. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  5. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  6. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  7. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 422
  8. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  9. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  10. 2024Stable Audio Opencited by 197
  11. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  12. 2023SoundStorm: Efficient Parallel Audio Generationcited by 114
  13. 2019Practical Lossless Compression with Latent Variables using Bits Back Codingcited by 86
  14. 2023VampNet: Music Generation via Masked Acoustic Token Modelingcited by 57
  15. 2024Sketch2Sound: Controllable Audio Generation via Time-Varying Signals and Sonic Imitationscited by 24
  16. 2025Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generationcited by 20
  17. 2025FLAM: Frame-Wise Language-Audio Modelingcited by 18
  18. 2017Fixing a Broken ELBOcited by 12
  19. 2020Interpreting Rate-Distortion of Variational Autoencoder and Using Model Uncertainty for Anomaly Detectioncited by 10
  20. 2025SpectroStream: A Versatile Neural Codec for General Audiocited by 4
  21. 2024SILA: Signal-to-Language Augmentation for Enhanced Control in Text-to-Audio Generationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning to Upsample and Upmix Audio in the Latent Domain
  2. 2026STAR-VAE: Structured Topology-Aware Regularization for Audio Reconstruction and Generation
  3. 2025Re-Bottleneck: Latent Re-Structuring for Neural Audio Autoencoders
  4. 2026Fast Text-to-Audio Generation with One-Step Sampling via Energy-Scoring and Auxiliary Contextual Representation Distillation ↗
  5. 2025Learning Interpretable Features in Audio Latent Spaces via Sparse Autoencoders
  6. 2023AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation
  7. 2026SwiftAudio: Data-Efficient Caption-Only Distillation for One-Step Text-to-Audio Diffusion-based Generation
  8. 2025Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementation