# Unified Latents (UL): How to train your latents

arXiv: https://arxiv.org/abs/2602.17270 · HTML: https://mlsanity.com/paper/2602.17270/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 16 of its 17 references that are in the map.
- First posted on arXiv 19 Feb 2026.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 16 of 16)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [Diffusion Autoencoders: Toward a Meaningful and Decodable Representation](https://mlsanity.com/paper/2111.15640/) (2021)
- [An Image is Worth 32 Tokens for Reconstruction and Generation](https://mlsanity.com/paper/2406.07550/) (2024)
- [Diffusion Transformers with Representation Autoencoders](https://mlsanity.com/paper/2510.11690/) (2025)
- [Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models](https://mlsanity.com/paper/2410.10733/) (2024)
- [DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents](https://mlsanity.com/paper/2201.00308/) (2022)
- [Latent Diffusion Model without Variational Autoencoder](https://mlsanity.com/paper/2510.15301/) (2025)
- [Preventing Posterior Collapse with delta-VAEs](https://mlsanity.com/paper/1901.03416/) (2019)
- [Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusion](https://mlsanity.com/paper/2410.19324/) (2024)
- [DiVAE: Photorealistic Images Synthesis with Denoising Diffusion Decoder](https://mlsanity.com/paper/2206.00386/) (2022)
- [Sample what you cant compress](https://mlsanity.com/paper/2409.02529/) (2024)

## Built on by (most cited first, 20 of 20)

- [Improved Baselines with Representation Autoencoders](https://mlsanity.com/paper/2605.18324/) (2026)
- [Making Reconstruction FID Predictive of Diffusion Generation FID](https://mlsanity.com/paper/2603.05630/) (2026)
- [What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion](https://mlsanity.com/paper/2605.07915/) (2026)
- [PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion](https://mlsanity.com/paper/2605.23902/) (2026)
- [Lens: Rethinking Training Efficiency for Foundational Text-to-Image Models](https://mlsanity.com/paper/2605.21573/) (2026)
- [Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation](https://mlsanity.com/paper/2607.27372/) (2026)
- [WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling](https://mlsanity.com/paper/2605.06407/) (2026)
- [SAME: A Semantically-Aligned Music Autoencoder](https://mlsanity.com/paper/2605.18613/) (2026)
- [WaiT for the Signal: Simple Frequency-Aware Flow-Matching](https://mlsanity.com/paper/2607.28760/) (2026)
- [MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generation](https://mlsanity.com/paper/2606.16408/) (2026)
- [CrossFlow: One-Step Generation Across Latent and Pixel Spaces](https://mlsanity.com/paper/2606.19970/) (2026)
- [GenFirst: Generation Before Reconstruction for Stable End-to-End Latent Generative Modeling](https://mlsanity.com/paper/2608.29335/) (2026)
- [Do multimodal models imagine electric sheep?](https://mlsanity.com/paper/2605.09693/) (2026)
- [FlashDecoder: Real-Time Latent-to-Pixel Streaming Decoder with Transformers](https://mlsanity.com/paper/2607.14898/) (2026)
- [Understanding Latent Diffusability via Fisher Geometry](https://mlsanity.com/paper/2604.02751/) (2026)
- [Provably Learning Diffusion Models under the Manifold Hypothesis: Collapse and Refine](https://mlsanity.com/paper/2605.20235/) (2026)
- [CenSynCMB: Centre Maps and Physics-Guided Synthesis for Microbleed Detection](https://mlsanity.com/paper/2607.05325/) (2026)
- [Weakly supervised concept Bottleneck Learning for Robust Two stage Object centric visual reasoning](https://mlsanity.com/paper/2608.22584/) (2026)
- [UNITE-AUDIO: Joint Learning of Continuous Tokenization and Latent Flow Matching for Text-to-Audio Generation](https://mlsanity.com/paper/2609.28206/) (2026)
- [MUNITE: Unified Multimodal Latent Inference for Any-to-Any Multimodal Generation](https://mlsanity.com/paper/2610.09866/) (2026)

## Abstract (arXiv)

We present Unified Latents (UL), a framework for learning latent representations that are jointly regularized by a diffusion prior and decoded by a diffusion model. By linking the encoder's output noise to the prior's minimum noise level, we obtain a simple training objective that provides a tight upper bound on the latent bitrate. On ImageNet-512, our approach achieves competitive FID of 1.4, with high reconstruction quality (PSNR) while requiring fewer training FLOPs than models trained on Stable Diffusion latents. On Kinetics-600, we set a new state-of-the-art FVD of 1.3.