# Information Theoretic Learning for Diffusion Models with Warm Start

arXiv: https://arxiv.org/abs/2510.20903 · HTML: https://mlsanity.com/paper/2510.20903/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 5 of its 94 references that are in the map.
- First posted on arXiv 23 Oct 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 5 of 5)

- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Lossy Compression with Gaussian Diffusion](https://mlsanity.com/paper/2206.08889/) (2022)
- [EVODiff: Entropy-aware Variance Optimized Diffusion Inference](https://mlsanity.com/paper/2509.26096/) (2025)
- [Generative Modeling with Bayesian Sample Inference](https://mlsanity.com/paper/2502.07580/) (2025)
- [Squeezed Diffusion Models](https://mlsanity.com/paper/2508.14871/) (2025)

## Built on by (most cited first, 4 of 4)

- [C2FG: Control Classifier-Free Guidance via Score Discrepancy Analysis](https://mlsanity.com/paper/2603.08155/) (2026)
- [A Minimum Variance Path Principle for Accurate and Stable Score-Based Density Ratio Estimation](https://mlsanity.com/paper/2602.00834/) (2026)
- [DiFA: Inference-Time Forward-Process Alignment for Diffusion Models](https://mlsanity.com/paper/2607.17972/) (2026)
- [Opening the Black Box of Classifier-Free Guidance via Information Bottleneck](https://mlsanity.com/paper/2610.13369/) (2026)

## Abstract (arXiv)

Generative models that maximize model likelihood have gained traction in many practical settings. Among them, perturbation based approaches underpin many strong likelihood estimation models, yet they often face slow convergence and limited theoretical understanding. In this paper, we derive a tighter likelihood bound for noise driven models to improve both the accuracy and efficiency of maximum likelihood learning. Our key insight extends the classical KL divergence Fisher information relationship to arbitrary noise perturbations, going beyond the Gaussian assumption and enabling structured noise distributions. This formulation allows flexible use of randomized noise distributions that naturally account for sensor artifacts, quantization effects, and data distribution smoothing, while remaining compatible with standard diffusion training. Treating the diffusion process as a Gaussian channel, we further express the mismatched entropy between data and model, showing that the proposed objective upper bounds the negative log-likelihood (NLL). In experiments, our models achieve competitive NLL on CIFAR-10 and SOTA results on ImageNet across multiple resolutions, all without data augmentation, and the framework extends naturally to discrete data.