mlsanityData to 7 Oct 2026

ControlMat: A Controlled Generative Approach to Material Capture

arXiv 2309.01700 ↗First posted 4 Sep 2023cs.CV cs.GRcited by 50

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Material reconstruction from a photograph is a key component of 3D content creation democratization. We propose to formulate this ill-posed problem as a controlled synthesis one, leveraging the recent progress in generative deep networks. We present ControlMat, a method which, given a single photograph with uncontrolled illumination as input, conditions a diffusion model to generate plausible, tileable, high-resolution physically-based digital materials. We carefully analyze the behavior of diffusion models for multi-channel outputs, adapt the sampling process to fuse multi-scale information and introduce rolled diffusion to enable both tileability and patched diffusion for high-resolution outputs. Our generative approach further permits exploration of a variety of materials which could correspond to the input image, mitigating the unknown lighting conditions. We show that our approach outperforms recent inference and latent-space-optimization methods, and carefully validate our diffusion process design choices. Supplemental materials and additional details are available at: https://gvecchio.com/controlmat/.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 44 of its 84 references that are in the map. First posted on arXiv 4 Sep 2023.

Builds on

44 of 84 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023MatFuse: Controllable Material Generation with Diffusion Models22 Aug

Other references · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
31 more
  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  3. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  6. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  7. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  8. 2017Improved Training of Wasserstein GANscited by 2,223
  9. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  10. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  11. 2021Designing a Practical Degradation Model for Deep Blind Image Super-Resolutioncited by 524
  12. 2016Generating Images with Perceptual Similarity Metrics based on Deep Networkscited by 221
  13. 2016Unrolled Generative Adversarial Networkscited by 208
  14. 2019Diagnosing and Enhancing VAE Modelscited by 192
  15. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  16. 2018Single-Image SVBRDF Capture with a Rendering-Aware Deep Networkcited by 116
  17. 2023Mixture of Diffusers for scene composition and high resolution image generationcited by 56
  18. 2018Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networkscited by 51
  19. 2019Flexible SVBRDF Capture with a Multi-Image Deep Networkcited by 50
  20. 2021Generative Modelling of BRDF Textures from Flash Imagescited by 35
  21. 2020MaterialGAN: Reflectance Capture using a Generative SVBRDF Modelcited by 33
  22. 2020Guided Fine-Tuning for Large-Scale Material Transfercited by 27
  23. 2022TileGen: Tileable, Controllable Material Generation and Capturecited by 27
  24. 2021SurfaceNet: Adversarial SVBRDF Estimation from a Single Imagecited by 25
  25. 2021An Inverse Procedural Modeling Pipeline for SVBRDF Mapscited by 24
  26. 2023Generating Procedural Materials from Text or Image Promptscited by 21
  27. 2023PhotoMat: A Material Generator Learned from Single Flash Photoscited by 18
  28. 2022Node Graph Optimization Using Differentiable Proxiescited by 13
  29. 2022MatFormer: A Generative Model for Procedural Materialscited by 12
  30. 2020Making Sense of CNNs: Interpreting Deep Representations & Their Invariances with INNscited by 10
  31. 2023Controlled and Conditional Text to Image Generation with Diffusion Priorcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 21; 2025: 21; 2026: 7.

50 papers.

Sort:

By area:

  1. 2024DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generationcited by 93
  2. 2024FlashTex: Fast Relightable Mesh Texturing with LightControlNetcited by 56
  3. 2024DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Modelscited by 50
  4. 2024MatSynth: A Modern PBR Materials Datasetcited by 42
  5. 2024Material Anything: Generating Materials for Any 3D Object via Diffusioncited by 31
  6. 2023Material Palette: Extraction of Materials from a Single Imagecited by 28
  7. 2024MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priorscited by 28
  8. 2024TexSliders: Diffusion-Based Texture Editing in CLIP Spacecited by 23
  9. 2024Collaborative Control for Geometry-Conditioned PBR Image Generationcited by 22
  10. 2024BlenderAlchemy: Editing 3D Graphics with Vision-Language Modelscited by 18
  11. 2024Boosting 3D Object Generation through PBR Materialscited by 10
  12. 2024MaterialPicker: Multi-Modal DiT-Based Material Generationcited by 10
38 more
  1. 2024SuperMat: Physically Consistent PBR Material Estimation at Interactive Ratescited by 9
  2. 2024SAMa: Material-aware 3D Selection and Segmentationcited by 9
  3. 2025VLMaterial: Procedural Material Generation with Large Vision-Language Modelscited by 9
  4. 2026Generative World Renderercited by 8
  5. 2024TexTile: A Differentiable Metric for Texture Tileabilitycited by 7
  6. 2025SViM3D: Stable Video Material Diffusion for Single Image 3D Generationcited by 7
  7. 2024MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D Assetscited by 6
  8. 2025MatSwap: Light-aware material transfers in imagescited by 6
  9. 2025Chord: Chain of Rendering Decomposition for PBR Material Estimation from Generated Texture Imagescited by 5
  10. 2024StableMaterials: Enhancing Diversity in Material Generation via Semi-Supervised Learningcited by 4
  11. 2024Uncertainty for SVBRDF Acquisition using Frequency Analysiscited by 4
  12. 2025Teamwork: Collaborative Diffusion with Low-rank Coordination and Adaptationcited by 3
  13. 2025MatPedia: A Universal Generative Foundation for High-Fidelity Material Synthesiscited by 3
  14. 2024DreamPBR: Text-driven Generation of High-resolution SVBRDF with Multi-modal Guidancecited by 2
  15. 2024Content-aware Tile Generation using Exterior Boundary Inpaintingcited by 2
  16. 2025MatCLIP: Light- and Shape-Insensitive Assignment of PBR Material Modelscited by 2
  17. 2025Generative Detail Enhancement for Physically Based Materialscited by 2
  18. 2025MaRI: Material Retrieval Integration across Domainscited by 2
  19. 2025A Controllable Appearance Representation for Flexible Transfer and Editingcited by 2
  20. 2026VideoNeuMat: Neural Material Extraction from Generative Video Modelscited by 2
  21. 2024FabricDiffusion: High-Fidelity Texture Transfer for 3D Garments Generation from In-The-Wild Clothing Imagescited by 1
  22. 2025Single-image Reflectance and Transmittance Estimation from Any Flatbed Scannercited by 1
  23. 2025DualMat: PBR Material Estimation via Coherent Dual-Path Diffusioncited by 1
  24. 2025RealMat: Realistic Materials with Diffusion and Reinforcement Learningcited by 1
  25. 2025Example-Based Feature Painting on Texturescited by 1
  26. 2026Reflecting Process Expertise in Procedural Material Generationcited by 1
  27. 2024Structured Pattern Expansion with Diffusion Modelscited by 0
  28. 2025HiMat: DiT-based Ultra-High Resolution SVBRDF Generationcited by 0
  29. 2025MUJICA: Reforming SISR Models for PBR Material Super-Resolution via Cross-Map Attentioncited by 0
  30. 2025SpecGen: Neural Spectral BRDF Generation via Spectral-Spatial Tri-plane Aggregationcited by 0
  31. 2025Phieat: Physically Grounded Material Feature Representationcited by 0
  32. 2025FROMAT: Multiview Material Appearance Transfer via Few-Shot Self-Attention Adaptationcited by 0
  33. 2025An evaluation of SVBRDF Prediction from Generative Image Models for Appearance Modeling of 3D Scenescited by 0
  34. 2025MatE: Material Extraction from Single-Image via Geometric Priorcited by 0
  35. 2026HeatMat: Simulation of City Material Impact on Urban Heat Island Effectcited by 0
  36. 2026FabricGen: Microstructure-Aware Woven Fabric Generationcited by 0
  37. 2026HAFMat: Hybrid Priors Guided Adaptive Fusion for Single-Image Human Material Estimationcited by 0
  38. 2026Texture Space Material Diffusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DealMaTe: Multi-Dimensional Material Transfer via Diffusion Transformer
  2. 2025MatMart: Material Reconstruction of 3D Objects via Diffusion
  3. 2024MatFusion: A Generative Diffusion Model for SVBRDF Capture
  4. 2026MaTe: Images Are All You Need for Material Transfer via Diffusion Transformer
  5. 2023Alchemist: Parametric Control of Material Properties with Diffusion Models
  6. 2023Intrinsic Image Diffusion for Indoor Single-view Material Estimation
  7. 2024IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination
  8. 2026Diffusion-Based Material Regularization for Physics-Based Inverse Rendering