mlsanityData to 14 Oct 2026

MatFuse: Controllable Material Generation with Diffusion Models

arXiv 2308.11408 ↗First posted 22 Aug 2023cs.CV cs.GRcited by 39

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Creating high-quality materials in computer graphics is a challenging and time-consuming task, which requires great expertise. To simplify this process, we introduce MatFuse, a unified approach that harnesses the generative power of diffusion models for creation and editing of 3D materials. Our method integrates multiple sources of conditioning, including color palettes, sketches, text, and pictures, enhancing creative possibilities and granting fine-grained control over material synthesis. Additionally, MatFuse enables map-level material editing capabilities through latent manipulation by means of a multi-encoder compression model which learns a disentangled latent representation for each map. We demonstrate the effectiveness of MatFuse under multiple conditioning settings and explore the potential of material editing. Finally, we assess the quality of the generated materials both quantitatively in terms of CLIP-IQA and FID scores and qualitatively by conducting a user study. Source code for training MatFuse and supplemental materials are publicly available at https://gvecchio.com/matfuse.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 37 of its 62 references that are in the map. First posted on arXiv 22 Aug 2023.

Builds on

37 of 62 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  10. 2022Classifier-Free Diffusion Guidancecited by 5,847
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  12. 2017Neural Discrete Representation Learningcited by 4,712
25 more
  1. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  2. 2014Going Deeper with Convolutionscited by 4,241
  3. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  5. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  6. 2016Improved Techniques for Training GANscited by 2,970
  7. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  8. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  9. 2017Improved Training of Wasserstein GANscited by 2,235
  10. 2014Conditional Generative Adversarial Netscited by 2,162
  11. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 991
  12. 2021Designing a Practical Degradation Model for Deep Blind Image Super-Resolutioncited by 529
  13. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 304
  14. 2016Generating Images with Perceptual Similarity Metrics based on Deep Networkscited by 222
  15. 2016Unrolled Generative Adversarial Networkscited by 208
  16. 2023Infinite Photorealistic Worlds using Procedural Generationcited by 147
  17. 2018Single-Image SVBRDF Capture with a Rendering-Aware Deep Networkcited by 116
  18. 2018Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Imagecited by 66
  19. 2020Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Imagescited by 65
  20. 2018Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networkscited by 51
  21. 2019Flexible SVBRDF Capture with a Multi-Image Deep Networkcited by 50
  22. 2022TileGen: Tileable, Controllable Material Generation and Capturecited by 27
  23. 2021SurfaceNet: Adversarial SVBRDF Estimation from a Single Imagecited by 25
  24. 2022Controlling Material Appearance by Examplescited by 10
  25. 2023Transformer-based Image Generation from Scene Graphscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 11; 2025: 19; 2026: 7.

39 papers.

Sort:

By area:

  1. 2025Hunyuan3D 2.5: Towards High-Fidelity 3D Assets Generation with Ultimate Detailscited by 125
  2. 2024FlashTex: Fast Relightable Mesh Texturing with LightControlNetcited by 56
  3. 2023ControlMat: A Controlled Generative Approach to Material Capturecited by 50
  4. 2024DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Modelscited by 50
  5. 2026SAGE: Scalable Agentic 3D Scene Generation for Embodied AIcited by 50
  6. 2024MatSynth: A Modern PBR Materials Datasetcited by 42
  7. 2025DRAWER: Digital Reconstruction and Articulation With Environment Realismcited by 29
  8. 2023Material Palette: Extraction of Materials from a Single Imagecited by 28
  9. 2025MaterialMVP: Illumination-Invariant Material Generation via Multi-view PBR Diffusioncited by 28
  10. 2024BlenderAlchemy: Editing 3D Graphics with Vision-Language Modelscited by 22
  11. 2024MaterialPicker: Multi-Modal DiT-Based Material Generationcited by 10
  12. 2026Generative World Renderercited by 10
27 more
  1. 2025VLMaterial: Procedural Material Generation with Large Vision-Language Modelscited by 9
  2. 2024MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D Assetscited by 6
  3. 2025Sampling 3D Molecular Conformers with Diffusion Transformerscited by 6
  4. 2025Chord: Chain of Rendering Decomposition for PBR Material Estimation from Generated Texture Imagescited by 5
  5. 2024StableMaterials: Enhancing Diversity in Material Generation via Semi-Supervised Learningcited by 4
  6. 2024Jointly Generating Multi-view Consistent PBR Textures using Collaborative Controlcited by 4
  7. 2025UnMix-NeRF: Spectral Unmixing Meets Neural Radiance Fieldscited by 3
  8. 2025MatPedia: A Universal Generative Foundation for High-Fidelity Material Synthesiscited by 3
  9. 2025MatCLIP: Light- and Shape-Insensitive Assignment of PBR Material Modelscited by 2
  10. 2025MaRI: Material Retrieval Integration across Domainscited by 2
  11. 2025LumiTex: Towards High-Fidelity PBR Texture Generation with Illumination Contextcited by 2
  12. 2026VideoNeuMat: Neural Material Extraction from Generative Video Modelscited by 2
  13. 2024Physics-aligned Schr"odinger bridgecited by 1
  14. 2024FabricDiffusion: High-Fidelity Texture Transfer for 3D Garments Generation from In-The-Wild Clothing Imagescited by 1
  15. 2025Single-image Reflectance and Transmittance Estimation from Any Flatbed Scannercited by 1
  16. 2025Channel-wise Noise Scheduled Diffusion for Inverse Rendering in Indoor Scenescited by 1
  17. 2025DualMat: PBR Material Estimation via Coherent Dual-Path Diffusioncited by 1
  18. 2025RealMat: Realistic Materials with Diffusion and Reinforcement Learningcited by 1
  19. 2024Structured Pattern Expansion with Diffusion Modelscited by 0
  20. 2025HiMat: DiT-based Ultra-High Resolution SVBRDF Generationcited by 0
  21. 2025MUJICA: Reforming SISR Models for PBR Material Super-Resolution via Cross-Map Attentioncited by 0
  22. 2025SpecGen: Neural Spectral BRDF Generation via Spectral-Spatial Tri-plane Aggregationcited by 0
  23. 2025MatE: Material Extraction from Single-Image via Geometric Priorcited by 0
  24. 2026HeatMat: Simulation of City Material Impact on Urban Heat Island Effectcited by 0
  25. 2026FabricGen: Microstructure-Aware Woven Fabric Generationcited by 0
  26. 2026Text-based Tactile Graphics Generation for the Visually Impairedcited by 0
  27. 2026MatLoom: Layered Text-to-Material Generation in a Compact Program Spacecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MaPa: Text-driven Photorealistic Material Painting for 3D Shapes
  2. 2025CoatFusion: Controllable Material Coating in Images
  3. 2025MaterialFusion: High-Quality, Zero-Shot, and Controllable Material Transfer with Diffusion Models
  4. 2026Toward Richer Material Generation via Procedural Data Enhancement
  5. 2023TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models
  6. 2023Single Mesh Diffusion Models with Field Latents for Texture Generation
  7. 2024Material Anything: Generating Materials for Any 3D Object via Diffusion
  8. 2024MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors