mlsanityData to 7 Oct 2026

Material Palette: Extraction of Materials from a Single Image

arXiv 2311.17060 ↗First posted 28 Nov 2023cs.CV cs.GRcited by 28

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a method to extract physically-based rendering (PBR) materials from a single real-world image. We do so in two steps: first, we map regions of the image to material concepts using a diffusion model, which allows the sampling of texture images resembling each material in the scene. Second, we benefit from a separate network to decompose the generated textures into Spatially Varying BRDFs (SVBRDFs), providing us with materials ready to be used in rendering applications. Our approach builds on existing synthetic material libraries with SVBRDF ground truth, but also exploits a diffusion-generated RGB texture dataset to allow generalization to new samples using unsupervised domain adaptation (UDA). Our contributions are thoroughly evaluated on synthetic and real-world datasets. We further demonstrate the applicability of our method for editing 3D scenes with materials estimated from real photographs. The code and models will be made open-source. Project page: https://astra-vision.github.io/MaterialPalette/

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 39 of its 69 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

39 of 69 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2023Segment Anythingcited by 8,158
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  11. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  12. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
27 more
  1. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  2. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  3. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  4. 2019Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Imagecited by 202
  5. 2019Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wildcited by 180
  6. 2018Single-Image SVBRDF Capture with a Rendering-Aware Deep Networkcited by 116
  7. 2019Neural Inverse Rendering of an Indoor Scene from a Single Imagecited by 101
  8. 2018CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Renderingcited by 85
  9. 2019Fast Spatially-Varying Indoor Lighting Estimationcited by 72
  10. 2022Physically-Based Editing of Indoor Scene Lighting from a Single Imagecited by 62
  11. 2021Deep Polarization Imaging for 3D shape and SVBRDF Acquisitioncited by 56
  12. 2019Unsupervised Learning for Intrinsic Image Decomposition from a Single Imagecited by 55
  13. 2017Learning Texture Manifolds with the Periodic Spatial GANcited by 53
  14. 2018Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networkscited by 51
  15. 2023ControlMat: A Controlled Generative Approach to Material Capturecited by 50
  16. 2023MatFuse: Controllable Material Generation with Diffusion Modelscited by 39
  17. 2015Direct Intrinsics: Learning Albedo-Shading Decomposition by Convolutional Regressioncited by 32
  18. 2020Guided Fine-Tuning for Large-Scale Material Transfercited by 27
  19. 2022Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistencycited by 26
  20. 2023Materialistic: Selecting Similar Materials in Imagescited by 26
  21. 2021SurfaceNet: Adversarial SVBRDF Estimation from a Single Imagecited by 25
  22. 2022DenseMTL: Cross-task Attention Mechanism for Dense Multi-task Learningcited by 25
  23. 2022PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenescited by 22
  24. 2020Deep SVBRDF Estimation on Real Materialscited by 15
  25. 2023UMat: Uncertainty-Aware Single Image High Resolution Material Capturecited by 11
  26. 2018An Approximate Shading Model with Detail Decomposition for Object Relightingcited by 5
  27. 2022Multi-view Gradient Consistency for SVBRDF Estimation of Complex Scenes under Natural Illuminationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 16; 2026: 5.

28 papers.

Sort:

By area:

  1. 2024ColorPeel: Color Prompt Learning with Diffusion Models via Color and Shape Disentanglementcited by 28
  2. 2024ZeST: Zero-Shot Material Transfer from a Single Imagecited by 24
  3. 2025LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scanscited by 21
  4. 2025IntrinsiX: High-Quality PBR Generation using Image Priorscited by 18
  5. 2024Multimodal Latent Diffusion Model for Complex Sewing Pattern Generationcited by 12
  6. 2024Boosting 3D Object Generation through PBR Materialscited by 10
  7. 2025MARBLE: Material Recomposition and Blending in CLIP-Spacecited by 9
  8. 2026Generative World Renderercited by 8
  9. 2025SViM3D: Stable Video Material Diffusion for Single Image 3D Generationcited by 7
  10. 2024MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D Assetscited by 6
  11. 2025MatSwap: Light-aware material transfers in imagescited by 6
  12. 2025Multi-view Surface Reconstruction Using Normal and Reflectance Cuescited by 5
16 more
  1. 2025Chord: Chain of Rendering Decomposition for PBR Material Estimation from Generated Texture Imagescited by 5
  2. 2024StableMaterials: Enhancing Diversity in Material Generation via Semi-Supervised Learningcited by 4
  3. 20253D-Generalist: Self-Improving Vision-Language-Action Models for Crafting 3D Worldscited by 3
  4. 2025MatPedia: A Universal Generative Foundation for High-Fidelity Material Synthesiscited by 3
  5. 2025Voxify3D: Pixel Art Meets Volumetric Renderingcited by 3
  6. 2025MaRI: Material Retrieval Integration across Domainscited by 2
  7. 2025InstructAttribute: Fine-grained Object Attributes editing with Instructioncited by 2
  8. 2024FabricDiffusion: High-Fidelity Texture Transfer for 3D Garments Generation from In-The-Wild Clothing Imagescited by 1
  9. 2026Lunar-G2R: Geometry-to-Reflectance Learning for High-Fidelity Lunar BRDF Estimationcited by 1
  10. 2025HiMat: DiT-based Ultra-High Resolution SVBRDF Generationcited by 0
  11. 2025LumiX: Structured and Coherent Text-to-Intrinsic Generationcited by 0
  12. 2025FROMAT: Multiview Material Appearance Transfer via Few-Shot Self-Attention Adaptationcited by 0
  13. 2025MatE: Material Extraction from Single-Image via Geometric Priorcited by 0
  14. 2026FMMC: Harnessing the Power of Foundation Models for Accurate Material Classificationcited by 0
  15. 2026MatReplace: A Reference-Free, Conditioning-Aligned Benchmark for Material Replacement in Interior Scenescited by 0
  16. 2026MatLoom: Layered Text-to-Material Generation in a Compact Program Spacecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors
  2. 2025VideoMat: Extracting PBR Materials from Video Diffusion Models
  3. 2024IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination
  4. 2025MatCLIP: Light- and Shape-Insensitive Assignment of PBR Material Models
  5. 2025DualMat: PBR Material Estimation via Coherent Dual-Path Diffusion
  6. 2024DreamPBR: Text-driven Generation of High-resolution SVBRDF with Multi-modal Guidance
  7. 2024MatFusion: A Generative Diffusion Model for SVBRDF Capture
  8. 2023Intrinsic Image Diffusion for Indoor Single-view Material Estimation