# Langevin Dynamics with Continuous Tempering for Training Deep Neural Networks

arXiv: https://arxiv.org/abs/1703.04379 · HTML: https://mlsanity.com/paper/1703.04379/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2017.

## Built on by (most cited first, 7 of 7)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [BayesDAG: Gradient-Based Posterior Inference for Causal Discovery](https://mlsanity.com/paper/2307.13917/) (2023)
- [Provable Convergence and Limitations of Geometric Tempering for Langevin Dynamics](https://mlsanity.com/paper/2410.09697/) (2024)
- [Calibrating AI Models for Wireless Communications via Conformal Prediction](https://mlsanity.com/paper/2212.07775/) (2022)
- [Accelerating Convergence in Global Non-Convex Optimization with Reversible Diffusion](https://mlsanity.com/paper/2305.11493/) (2023)
- [Application of Langevin Dynamics to Advance the Quantum Natural Gradient Optimization Algorithm](https://mlsanity.com/paper/2409.01978/) (2024)
- [Fokker-Planck to Callan-Symanzik: evolution of weight matrices under training](https://mlsanity.com/paper/2501.09659/) (2025)

## Abstract (arXiv)

Minimizing non-convex and high-dimensional objective functions is challenging, especially when training modern deep neural networks. In this paper, a novel approach is proposed which divides the training process into two consecutive phases to obtain better generalization performance: Bayesian sampling and stochastic optimization. The first phase is to explore the energy landscape and to capture the "fat" modes; and the second one is to fine-tune the parameter learned from the first phase. In the Bayesian learning phase, we apply continuous tempering and stochastic approximation into the Langevin dynamics to create an efficient and effective sampler, in which the temperature is adjusted automatically according to the designed "temperature dynamics". These strategies can overcome the challenge of early trapping into bad local minima and have achieved remarkable improvements in various types of neural networks as shown in our theoretical analysis and empirical experiments.