# Beyond Surface Statistics: Scene Representations in a Latent Diffusion Model

arXiv: https://arxiv.org/abs/2306.05720 · HTML: https://mlsanity.com/paper/2306.05720/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 27 of its 32 references that are in the map.
- First posted on arXiv 9 Jun 2023.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation](https://mlsanity.com/paper/2212.00774/) (2022)
- [Label-Efficient Semantic Segmentation with Diffusion Models](https://mlsanity.com/paper/2112.03126/) (2021)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Experience Grounds Language](https://mlsanity.com/paper/2004.10151/) (2020)
- [Self-supervised Monocular Trained Depth Estimation using Self-attention and Discrete Disparity Volume](https://mlsanity.com/paper/2003.13951/) (2020)
- [Do Neural Language Representations Learn Physical Commonsense?](https://mlsanity.com/paper/1908.02899/) (2019)
- [What if This Modified That? Syntactic Interventions via Counterfactual Embeddings](https://mlsanity.com/paper/2105.14002/) (2021)
- [TRACER: Extreme Attention Guided Salient Object Tracing Network](https://mlsanity.com/paper/2112.07380/) (2021)
- [Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis](https://mlsanity.com/paper/1809.10040/) (2018)
- [Analyzing analytical methods: The case of phonology in neural models of spoken language](https://mlsanity.com/paper/2004.07070/) (2020)
- [DAG: Depth-Aware Guidance with Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2212.08861/) (2022)
- [GeoFill: Reference-Based Image Inpainting with Better Geometric Understanding](https://mlsanity.com/paper/2201.08131/) (2022)

## Built on by (most cited first, 30 of 30)

- [Probing the 3D Awareness of Visual Foundation Models](https://mlsanity.com/paper/2404.08636/) (2024)
- [Shadows Don't Lie and Lines Can't Bend! Generative Models don't know Projective Geometry...for now](https://mlsanity.com/paper/2311.17138/) (2023)
- [Generative Models: What Do They Know? Do They Know Things? Let's Find Out!](https://mlsanity.com/paper/2311.17137/) (2023)
- [From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit](https://mlsanity.com/paper/2506.03093/) (2025)
- [REEF: Representation Encoding Fingerprints for Large Language Models](https://mlsanity.com/paper/2410.14273/) (2024)
- [Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space](https://mlsanity.com/paper/2406.19370/) (2024)
- [The 3D-PC: a benchmark for visual perspective taking in humans and machines](https://mlsanity.com/paper/2406.04138/) (2024)
- [SIGNeRF: Scene Integrated Generation for Neural Radiance Fields](https://mlsanity.com/paper/2401.01647/) (2024)
- [Deciphering 'What' and 'Where' Visual Pathways from Spectral Clustering of Layer-Distributed Neural Representations](https://mlsanity.com/paper/2312.06716/) (2023)
- [Disrupting Diffusion-based Inpainters with Semantic Digression](https://mlsanity.com/paper/2407.10277/) (2024)
- [Viewpoint Textual Inversion: Discovering Scene Representations and 3D View Control in 2D Diffusion Models](https://mlsanity.com/paper/2309.07986/) (2023)
- [Detecting Origin Attribution for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2403.19653/) (2024)
- [Conceptual Framework for Autonomous Cognitive Entities](https://mlsanity.com/paper/2310.06775/) (2023)
- [Interpreting Large Text-to-Image Diffusion Models with Dictionary Learning](https://mlsanity.com/paper/2505.24360/) (2025)
- [Enhancing Multiple Dimensions of Trustworthiness in LLMs via Sparse Activation Control](https://mlsanity.com/paper/2411.02461/) (2024)
- [On LLMs' Internal Representation of Code Correctness](https://mlsanity.com/paper/2512.07404/) (2025)
- [Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts in Underspecified Visual Tasks](https://mlsanity.com/paper/2310.02230/) (2023)
- [Mitigating Shortcut Learning with Diffusion Counterfactuals and Diverse Ensembles](https://mlsanity.com/paper/2311.16176/) (2023)
- [Latent Abstractions in Generative Diffusion Models](https://mlsanity.com/paper/2410.03368/) (2024)
- [Unsupervised Segmentation by Diffusing, Walking and Cutting](https://mlsanity.com/paper/2412.04678/) (2024)
- [Latent Diffusion U-Net Representations Contain Positional Embeddings and Anomalies](https://mlsanity.com/paper/2504.07008/) (2025)
- [Heeding the Inner Voice: Aligning ControlNet Training via Intermediate Features Feedback](https://mlsanity.com/paper/2507.02321/) (2025)
- [SeeSE3: Emergence of 3D Space in Vision Features](https://mlsanity.com/paper/2607.14228/) (2026)
- [The Camera Inside the Editor: Reading the Implicit Camera of Image Editors with Painted Calibration Patterns](https://mlsanity.com/paper/2609.37732/) (2026)
- [NeuSD: Surface Completion with Multi-View Text-to-Image Diffusion](https://mlsanity.com/paper/2312.04654/) (2023)
- [DiffusionBrowser: Interactive Diffusion Previews via Multi-Branch Decoders](https://mlsanity.com/paper/2512.13690/) (2025)
- [An evaluation of SVBRDF Prediction from Generative Image Models for Appearance Modeling of 3D Scenes](https://mlsanity.com/paper/2512.13950/) (2025)
- [Interpretable Human-Label-Free Deep Learning for Real-Bogus Classification with Uncertainty Quantification](https://mlsanity.com/paper/2607.05393/) (2026)
- [LoRango: It Takes Two LoRAs to Unlock Hidden Behaviors in Diffusion Models](https://mlsanity.com/paper/2609.25884/) (2026)
- [Beyond the Linear Representation Hypothesis: Non-Linear Activation Steering in Text-to-Image Models](https://mlsanity.com/paper/2610.06945/) (2026)

## Abstract (arXiv)

Latent diffusion models (LDMs) exhibit an impressive ability to produce realistic images, yet the inner workings of these models remain mysterious. Even when trained purely on images without explicit depth information, they typically output coherent pictures of 3D scenes. In this work, we investigate a basic interpretability question: does an LDM create and use an internal representation of simple scene geometry? Using linear probes, we find evidence that the internal activations of the LDM encode linear representations of both 3D depth data and a salient-object / background distinction. These representations appear surprisingly early in the denoising process-well before a human can easily make sense of the noisy images. Intervention experiments further indicate these representations play a causal role in image synthesis, and may be used for simple high-level editing of an LDM's output. Project page: https://yc015.github.io/scene-representation-diffusion-model/