# SODA: Bottleneck Diffusion Models for Representation Learning

arXiv: https://arxiv.org/abs/2311.17901 · HTML: https://mlsanity.com/paper/2311.17901/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 91 of its 131 references that are in the map.
- First posted on arXiv 29 Nov 2023.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 91)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)

## Built on by (most cited first, 30 of 77)

- [Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think](https://mlsanity.com/paper/2410.06940/) (2024)
- [Deconstructing Denoising Diffusion Models for Self-Supervised Learning](https://mlsanity.com/paper/2401.14404/) (2024)
- [What matters for Representation Alignment: Global Information or Spatial Structure?](https://mlsanity.com/paper/2512.10794/) (2025)
- [Diffusion Models and Representation Learning: A Survey](https://mlsanity.com/paper/2407.00783/) (2024)
- [Schr"odinger Bridge Flow for Unpaired Data Translation](https://mlsanity.com/paper/2409.09347/) (2024)
- [Parameter-Efficient Fine-Tuning for Foundation Models](https://mlsanity.com/paper/2501.13787/) (2025)
- [CleanDIFT: Diffusion Features without Noise](https://mlsanity.com/paper/2412.03439/) (2024)
- [Diffusion Feedback Helps CLIP See Better](https://mlsanity.com/paper/2407.20171/) (2024)
- [Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenization](https://mlsanity.com/paper/2503.11056/) (2025)
- [Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement](https://mlsanity.com/paper/2402.09712/) (2024)
- [Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training](https://mlsanity.com/paper/2402.14407/) (2024)
- [USP: Unified Self-Supervised Pretraining for Image Generation and Understanding](https://mlsanity.com/paper/2503.06132/) (2025)
- [UniGEM: A Unified Approach to Generation and Property Prediction for Molecules](https://mlsanity.com/paper/2410.10516/) (2024)
- [SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulation](https://mlsanity.com/paper/2405.19586/) (2024)
- [Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling](https://mlsanity.com/paper/2502.05743/) (2025)
- [Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation](https://mlsanity.com/paper/2410.02369/) (2024)
- [GenHancer: Imperfect Generative Models are Secretly Strong Vision-Centric Enhancers](https://mlsanity.com/paper/2503.19480/) (2025)
- [MacDiff: Unified Skeleton Modeling with Masked Conditional Diffusion](https://mlsanity.com/paper/2409.10473/) (2024)
- [Image Understanding Makes for A Good Tokenizer for Image Generation](https://mlsanity.com/paper/2411.04406/) (2024)
- [Detect-and-Guide: Self-regulation of Diffusion Models for Safe Text-to-Image Generation via Guideline Token Optimization](https://mlsanity.com/paper/2503.15197/) (2025)
- [Visual Lexicon: Rich Image Features in Language Space](https://mlsanity.com/paper/2412.06774/) (2024)
- [Visual Representation Learning with Stochastic Frame Prediction](https://mlsanity.com/paper/2406.07398/) (2024)
- [Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation](https://mlsanity.com/paper/2412.04432/) (2024)
- [D-AR: Diffusion via Autoregressive Models](https://mlsanity.com/paper/2505.23660/) (2025)
- [MotionStrata: Hierarchical Motion Latents for Compact Video Autoencoding](https://mlsanity.com/paper/2506.07136/) (2025)
- [4D Visual Pre-training for Robot Learning](https://mlsanity.com/paper/2508.17230/) (2025)
- [Understanding the Role of Equivariance in Self-supervised Learning](https://mlsanity.com/paper/2411.06508/) (2024)
- [Unleashing the Potential of Pre-Trained Diffusion Models for Generalizable Person Re-Identification](https://mlsanity.com/paper/2502.06619/) (2025)
- [SSDD: Single-Step Diffusion Decoder for Efficient Image Tokenization](https://mlsanity.com/paper/2510.04961/) (2025)
- [Diffusion Bridge AutoEncoders for Unsupervised Representation Learning](https://mlsanity.com/paper/2405.17111/) (2024)

## Abstract (arXiv)

We introduce SODA, a self-supervised diffusion model, designed for representation learning. The model incorporates an image encoder, which distills a source view into a compact representation, that, in turn, guides the generation of related novel views. We show that by imposing a tight bottleneck between the encoder and a denoising decoder, and leveraging novel view synthesis as a self-supervised objective, we can turn diffusion models into strong representation learners, capable of capturing visual semantics in an unsupervised manner. To the best of our knowledge, SODA is the first diffusion model to succeed at ImageNet linear-probe classification, and, at the same time, it accomplishes reconstruction, editing and synthesis tasks across a wide range of datasets. Further investigation reveals the disentangled nature of its emergent latent space, that serves as an effective interface to control and manipulate the model's produced images. All in all, we aim to shed light on the exciting and promising potential of diffusion models, not only for image generation, but also for learning rich and robust representations.