mlsanityData to 13 Oct 2026

Beyond Surface Statistics: Scene Representations in a Latent Diffusion Model

arXiv 2306.05720 ↗First posted 9 Jun 2023cs.CV cs.AI cs.LGcited by 30

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Latent diffusion models (LDMs) exhibit an impressive ability to produce realistic images, yet the inner workings of these models remain mysterious. Even when trained purely on images without explicit depth information, they typically output coherent pictures of 3D scenes. In this work, we investigate a basic interpretability question: does an LDM create and use an internal representation of simple scene geometry? Using linear probes, we find evidence that the internal activations of the LDM encode linear representations of both 3D depth data and a salient-object / background distinction. These representations appear surprisingly early in the denoising process-well before a human can easily make sense of the noisy images. Intervention experiments further indicate these representations play a causal role in image synthesis, and may be used for simple high-level editing of an LDM's output. Project page: https://yc015.github.io/scene-representation-diffusion-model/

Key facts. Cited by 30 papers in the map (data through 13 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 27 of its 32 references that are in the map. First posted on arXiv 9 Jun 2023.

Builds on

27 of 32 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022DAG: Depth-Aware Guidance with Denoising Diffusion Probabilistic Models17 Dec
  2. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  3. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 24

  1. 2017Attention Is All You Needcited by 60,106
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,952
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  6. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,023
  7. 2019Language Models as Knowledge Bases?cited by 1,931
  8. 2021Vision Transformers for Dense Predictioncited by 1,795
  9. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,548
  10. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,490
  11. 2016Understanding intermediate layers using linear classifier probescited by 1,313
  12. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,269
12 more
  1. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,049
  2. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 496
  3. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 455
  4. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 421
  5. 2020Experience Grounds Languagecited by 241
  6. 2020Self-supervised Monocular Trained Depth Estimation using Self-attention and Discrete Disparity Volumecited by 116
  7. 2019Do Neural Language Representations Learn Physical Commonsense?cited by 62
  8. 2021What if This Modified That? Syntactic Interventions via Counterfactual Embeddingscited by 39
  9. 2021TRACER: Extreme Attention Guided Salient Object Tracing Networkcited by 32
  10. 2018Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysiscited by 17
  11. 2020Analyzing analytical methods: The case of phonology in neural models of spoken languagecited by 16
  12. 2022GeoFill: Reference-Based Image Inpainting with Better Geometric Understandingcited by 9

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 8; 2024: 10; 2025: 7; 2026: 5.

30 papers.

Sort:

By area:

  1. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 176
  2. 2023Shadows Don't Lie and Lines Can't Bend! Generative Models don't know Projective Geometry...for nowcited by 61
  3. 2023Generative Models: What Do They Know? Do They Know Things? Let's Find Out!cited by 50
  4. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 48
  5. 2024REEF: Representation Encoding Fingerprints for Large Language Modelscited by 36
  6. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 34
  7. 2024The 3D-PC: a benchmark for visual perspective taking in humans and machinescited by 16
  8. 2024SIGNeRF: Scene Integrated Generation for Neural Radiance Fieldscited by 11
  9. 2023Deciphering 'What' and 'Where' Visual Pathways from Spectral Clustering of Layer-Distributed Neural Representationscited by 9
  10. 2024Disrupting Diffusion-based Inpainters with Semantic Digressioncited by 8
  11. 2023Viewpoint Textual Inversion: Discovering Scene Representations and 3D View Control in 2D Diffusion Modelscited by 7
  12. 2024Detecting Origin Attribution for Text-to-Image Diffusion Modelscited by 7
18 more
  1. 2023Conceptual Framework for Autonomous Cognitive Entitiescited by 6
  2. 2025Interpreting Large Text-to-Image Diffusion Models with Dictionary Learningcited by 6
  3. 2024Enhancing Multiple Dimensions of Trustworthiness in LLMs via Sparse Activation Controlcited by 5
  4. 2025On LLMs' Internal Representation of Code Correctnesscited by 5
  5. 2023Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts in Underspecified Visual Taskscited by 4
  6. 2023Mitigating Shortcut Learning with Diffusion Counterfactuals and Diverse Ensemblescited by 4
  7. 2024Latent Abstractions in Generative Diffusion Modelscited by 4
  8. 2024Unsupervised Segmentation by Diffusing, Walking and Cuttingcited by 3
  9. 2025Latent Diffusion U-Net Representations Contain Positional Embeddings and Anomaliescited by 1
  10. 2025Heeding the Inner Voice: Aligning ControlNet Training via Intermediate Features Feedbackcited by 1
  11. 2026SeeSE3: Emergence of 3D Space in Vision Featurescited by 1
  12. 2026The Camera Inside the Editor: Reading the Implicit Camera of Image Editors with Painted Calibration Patternscited by 1
  13. 2023NeuSD: Surface Completion with Multi-View Text-to-Image Diffusioncited by 0
  14. 2025DiffusionBrowser: Interactive Diffusion Previews via Multi-Branch Decoderscited by 0
  15. 2025An evaluation of SVBRDF Prediction from Generative Image Models for Appearance Modeling of 3D Scenescited by 0
  16. 2026Interpretable Human-Label-Free Deep Learning for Real-Bogus Classification with Uncertainty Quantificationcited by 0
  17. 2026LoRango: It Takes Two LoRAs to Unlock Hidden Behaviors in Diffusion Modelscited by 0
  18. 2026Beyond the Linear Representation Hypothesis: Non-Linear Activation Steering in Text-to-Image Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LT3SD: Latent Trees for 3D Scene Diffusion
  2. 2026Predicting 3D structure by latent posterior sampling
  3. 2024Denoising Diffusion via Image-Based Rendering
  4. 2024Diffusion Models in 3D Vision: A Survey
  5. 2024Diffusion Models in Low-Level Vision: A Survey
  6. 2025From Image to Video: An Empirical Study of Diffusion Representations
  7. 2026The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show
  8. 2025Towards Spatially Consistent Image Generation: On Incorporating Intrinsic Scene Properties into Diffusion Models