mlsanityData to 7 Oct 2026

Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing

arXiv 1903.10145 ↗First posted 25 Mar 2019cs.LG cs.AI cs.CLcited by 191

Not on the map. Why?

Abstract (arXiv)

Variational autoencoders (VAEs) with an auto-regressive decoder have been applied for many natural language processing (NLP) tasks. The VAE objective consists of two terms, (i) reconstruction and (ii) KL regularization, balanced by a weighting hyper-parameter β. One notorious training difficulty is that the KL term tends to vanish. In this paper we study scheduling schemes for β, and show that KL vanishing is caused by the lack of good latent codes in training the decoder at the beginning of optimization. To remedy this, we propose a cyclical annealing schedule, which repeats the process of increasing β multiple times. This new procedure allows the progressive learning of more meaningful latent codes, by leveraging the informative representations of previous cycles as warm re-starts. The effectiveness of cyclical annealing is validated on a broad range of NLP tasks, including language modeling, dialog response generation and unsupervised language pre-training.

Key facts. Cited by 191 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 41; 2022: 36; 2023: 40; 2024: 26; 2025: 25; 2026: 23.

191 papers.

Sort:

By area:

  1. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  2. 2022Latent Diffusion Energy-Based Model for Interpretable Text Modelingcited by 67
  3. 2022NeMF: Neural Motion Fields for Kinematic Animationcited by 66
  4. 2023Posterior Collapse and Latent Variable Non-identifiabilitycited by 57
  5. 2021Beyond Trivial Counterfactual Explanations with Diverse Valuable Explanationscited by 56
  6. 2021Consistency Regularization for Variational Auto-Encoderscited by 50
  7. 2021Transformer-based Conditional Variational Autoencoder for Controllable Story Generationcited by 44
  8. 2021MuseMorphose: Full-Song and Fine-Grained Piano Music Style Transfer with One Transformer VAEcited by 39
  9. 2022Geometry Aligned Variational Transformer for Image-conditioned Layout Generationcited by 33
  10. 2023Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecastingcited by 33
  11. 2023Latent Feature-Guided Diffusion Models for Shadow Removalcited by 28
  12. 2021Annealed Stein Variational Gradient Descentcited by 24
38 more
  1. 2021Syntactically Guided Generative Embeddings for Zero-Shot Skeleton Action Recognitioncited by 24
  2. 2021Convex Aggregation for Opinion Summarizationcited by 21
  3. 2021On Stein Variational Neural Network Ensemblescited by 21
  4. 2023BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformercited by 21
  5. 2021Generative Chemical Transformer: Neural Machine Learning of Molecular Geometric Structures from Chemical Language via Attentioncited by 20
  6. 2022Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generationcited by 20
  7. 2023Slot-VAE: Object-Centric Scene Generation with Slot Attentioncited by 20
  8. 2022A Personalized Dialogue Generator with Implicit User Persona Detectioncited by 19
  9. 2024SA-DVAE: Improving Zero-Shot Skeleton-Based Action Recognition by Disentangled Variational Autoencoderscited by 19
  10. 2022Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curvecited by 18
  11. 2021Variational Topic Inference for Chest X-Ray Report Generationcited by 16
  12. 2023Latent SDEs on Homogeneous Spacescited by 16
  13. 2021Preventing Oversmoothing in VAE via Generalized Variance Parameterizationcited by 15
  14. 2023A Framework for Demonstrating Practical Quantum Advantage: Racing Quantum against Classical Generative Modelscited by 15
  15. 2021Finetuning Pretrained Transformers into Variational Autoencoderscited by 14
  16. 2021Latent Space Energy-Based Model of Symbol-Vector Coupling for Text Generation and Classificationcited by 14
  17. 2021MUGL: Large Scale Multi Person Conditional Action Generation with Locomotioncited by 14
  18. 2022Hierarchical Latent Structure for Multi-Modal Vehicle Trajectory Forecastingcited by 14
  19. 2021Controllable and Diverse Text Generation in E-commercecited by 13
  20. 2021Acceleration via Fractal Learning Rate Schedulescited by 13
  21. 2023RegaVAE: A Retrieval-Augmented Gaussian Mixture Variational Auto-Encoder for Language Modelingcited by 12
  22. 2023Learning Dynamics in Linear VAE: Posterior Collapse Threshold, Superfluous Latent Space Pitfalls, and Speedup with KL Annealingcited by 12
  23. 2021Exploring Social Posterior Collapse in Variational Autoencoder for Interaction Modelingcited by 11
  24. 2023Lewis's Signaling Game as beta-VAE For Natural Word Lengths and Segmentscited by 11
  25. 2024Poisson Variational Autoencodercited by 11
  26. 2021Provable Super-Convergence with a Large Cyclical Learning Ratecited by 10
  27. 2022AdaVAE: Exploring Adaptive GPT-2s in Variational Auto-Encoders for Language Modelingcited by 10
  28. 2023Explainable Representation Learning of Small Quantum Statescited by 10
  29. 2024Unified Generation, Reconstruction, and Representation: Generalized Diffusion with Adaptive Latent Encoding-Decodingcited by 10
  30. 2021Variational Knowledge Distillation for Disease Classification in Chest X-Rayscited by 9
  31. 2022Domain Knowledge Driven Pseudo Labels for Interpretable Goal-Conditioned Interactive Trajectory Predictioncited by 9
  32. 2022Hierarchically Structured Task-Agnostic Continual Learningcited by 9
  33. 2024DiffLM: Controllable Synthetic Data Generation via Diffusion Language Modelscited by 9
  34. 2021An Exploration of Learnt Representations of W Jetscited by 8
  35. 2021Learning Continuous Environment Fields via Implicit Functionscited by 8
  36. 2022SmartMocap: Joint Estimation of Human and Camera Motion using Uncalibrated RGB Camerascited by 8
  37. 2022Prompt Conditioned VAE: Enhancing Generative Replay for Lifelong Learning in Task-Oriented Dialoguecited by 8
  38. 2023SAiD: Speech-driven Blendshape Facial Animation with Diffusioncited by 8

50 most cited of 191 All 191 (JSON)