mlsanityData to 14 Oct 2026

StyleGAN knows Normal, Depth, Albedo, and More

arXiv 2306.00987 ↗First posted 1 Jun 2023cs.CV cs.GR cs.LGcited by 44

Not on the map. Why?

Abstract (arXiv)

Intrinsic images, in the original sense, are image-like maps of scene properties like depth, normal, albedo or shading. This paper demonstrates that StyleGAN can easily be induced to produce intrinsic images. The procedure is straightforward. We show that, if StyleGAN produces G(w) from latents w, then for each type of intrinsic image, there is a fixed offset dc so that G(w+dc) is that type of intrinsic image for G(w). Here dc is em independent of w. The StyleGAN we used was pretrained by others, so this property is not some accident of our training regime. We show that there are image transformations StyleGAN will em not produce in this fashion, so StyleGAN is not a generic image regression engine. It is conceptually exciting that an image generator should ``know'' and represent intrinsic images. There may also be practical advantages to using a generative model to produce intrinsic images. The intrinsic images obtained from StyleGAN compare well both qualitatively and quantitatively with those obtained by using SOTA image regression techniques; but StyleGAN's intrinsic images are robust to relighting effects, unlike SOTA methods.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Builds on 66 of its 78 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

66 of 78 references.

References · 66

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  4. 2022Generative Adversarial Networkscited by 5,627
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  8. 2016Improved Techniques for Training GANscited by 2,970
  9. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 2,010
  10. 2021Vision Transformers for Dense Predictioncited by 1,803
  11. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  12. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
38 more
  1. 2021Alias-Free Generative Adversarial Networkscited by 1,184
  2. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  3. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  4. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  5. 2019An Introduction to Variational Autoencoderscited by 1,000
  6. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  7. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  8. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 657
  9. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  10. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  11. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  12. 2020Closed-Form Factorization of Latent Semantics in GANscited by 408
  13. 2020InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANscited by 394
  14. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  15. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 375
  16. 2020StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flowscited by 349
  17. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  18. 2020StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generationcited by 332
  19. 2020Unsupervised Discovery of Interpretable Directions in the GAN Latent Spacecited by 289
  20. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 269
  21. 2021DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effortcited by 264
  22. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 255
  23. 2022Visual Prompting via Image Inpaintingcited by 250
  24. 2021HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editingcited by 208
  25. 2021High-Fidelity GAN Inversion for Image Attribute Editingcited by 197
  26. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clonescited by 172
  27. 2021EditGAN: High-Precision Semantic Image Editingcited by 156
  28. 2021Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalizationcited by 141
  29. 20223D Common Corruptions and Data Augmentationcited by 119
  30. 2019Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesiscited by 106
  31. 2020Robust Learning Through Cross-Task Consistencycited by 105
  32. 2021Generative Models as a Data Source for Multiview Representation Learningcited by 104
  33. 2020Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Renderingcited by 101
  34. 2018InverseRenderNet: Learning single image inverse renderingcited by 100
  35. 2021GAN-Control: Explicitly Controllable GANscited by 91
  36. 2021A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesiscited by 72
  37. 2020Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANscited by 63
  38. 2019Unsupervised Learning for Intrinsic Image Decomposition from a Single Imagecited by 55

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 16; 2025: 13; 2026: 8.

44 papers.

Sort:

By area:

  1. 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 327
  2. 2024GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Imagecited by 286
  3. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 177
  4. 2025Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysiscited by 78
  5. 2024What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?cited by 74
  6. 2023Intrinsic Image Diffusion for Indoor Single-view Material Estimationcited by 73
  7. 2023Shadows Don't Lie and Lines Can't Bend! Generative Models don't know Projective Geometry...for nowcited by 61
  8. 2025DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Modelscited by 57
  9. 2023Generative Models: What Do They Know? Do They Know Things? Let's Find Out!cited by 50
  10. 2026Image Generators are Generalist Vision Learnerscited by 43
  11. 2025UniRelight: Learning Joint Decomposition and Synthesis for Video Relightingcited by 40
  12. 2023Exploiting Diffusion Prior for Generalizable Dense Predictioncited by 38
32 more
  1. 2024LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relightingcited by 38
  2. 2023A General Protocol to Probe Large Vision Models for 3D Physical Understandingcited by 32
  3. 2024Videoshop: Localized Semantic Video Editing with Noise-Extrapolated Diffusion Inversioncited by 27
  4. 2024ScribbleLight: Single Image Indoor Relighting with Scribblescited by 24
  5. 2023UrbanIR: Large-Scale Urban Scene Inverse Rendering from a Single Videocited by 22
  6. 2024GenLit: Reformulating Single-Image Relighting as Video Generationcited by 20
  7. 2024Latent Intrinsics Emerge from Training to Relightcited by 17
  8. 2024The 3D-PC: a benchmark for visual perspective taking in humans and machinescited by 16
  9. 2024ZeroComp: Zero-shot Object Compositing from Image Intrinsics via Diffusioncited by 16
  10. 2025Ouroboros: Single-step Diffusion Models for Cycle-consistent Forward and Inverse Renderingcited by 16
  11. 2024Multiview Equivariance Improves 3D Correspondence Understanding with Minimal Feature Finetuningcited by 14
  12. 2024NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFscited by 11
  13. 2025Visual Jenga: Discovering Object Dependencies via Counterfactual Inpaintingcited by 11
  14. 2025PRISM: A Unified Framework for Photorealistic Reconstruction and Intrinsic Scene Modelingcited by 9
  15. 2024Boost 3D Reconstruction using Diffusion-based Monocular Camera Calibrationcited by 8
  16. 2025Frozen Forecasting: A Unified Evaluationcited by 4
  17. 2025OmniX: From Unified Panoramic Generation and Perception to Graphics-Ready 3D Scenescited by 4
  18. 2025From Ideal to Real: Unified and Data-Efficient Dense Prediction for Real-World Scenarioscited by 3
  19. 2024Zippo: Zipping Color and Transparency Distributions into a Single Diffusion Modelcited by 2
  20. 2024Composing Parts for Expressive Object Generationcited by 2
  21. 2025gen2seg: Generative Models Enable Generalizable Instance Segmentationcited by 2
  22. 2025Generative Blocks World: Moving Things Around in Picturescited by 2
  23. 2026Principia: Relational Physics Tests for Video Modelscited by 2
  24. 2026ICTPolarReal: A Polarized Reflection and Material Dataset of Real World Objectscited by 1
  25. 2026SeeSE3: Emergence of 3D Space in Vision Featurescited by 1
  26. 2024Probabilistic Directed Distance Fields for Ray-Based Shape Representationscited by 0
  27. 2025LumiX: Structured and Coherent Text-to-Intrinsic Generationcited by 0
  28. 2025An evaluation of SVBRDF Prediction from Generative Image Models for Appearance Modeling of 3D Scenescited by 0
  29. 2026SyncLight: Single-Edit Multi-View Relightingcited by 0
  30. 2026Relighting as a Probe of Visual Priors via Augmented Latent Intrinsicscited by 0
  31. 2026Lighting-Consistent Object Transfer Across Radiance Fieldscited by 0
  32. 2026FracGen: Learning How Objects Stretch and Tear with Physics-Informed Video Generationcited by 0