mlsanityData to 14 Oct 2026

Alchemist: Parametric Control of Material Properties with Diffusion Models

arXiv 2312.02970 ↗First posted 5 Dec 2023cs.CV cs.AI cs.GRcited by 38

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We propose a method to control material attributes of objects like roughness, metallic, albedo, and transparency in real images. Our method capitalizes on the generative prior of text-to-image models known for photorealism, employing a scalar value and instructions to alter low-level material properties. Addressing the lack of datasets with controlled material attributes, we generated an object-centric synthetic dataset with physically-based materials. Fine-tuning a modified pre-trained text-to-image model on this synthetic dataset enables us to edit material properties in real-world images while preserving all other attributes. We show the potential application of our model to material edited NeRFs.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 71 of its 98 references that are in the map. First posted on arXiv 5 Dec 2023.

Builds on

71 of 98 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  2. 2023IT3D: Improved Text-to-3D Generation with Explicit View Synthesis22 Aug
  3. 2023Edit-DiffNeRF: Editing 3D Neural Radiance Fields using 2D Diffusion Model15 Jun
  4. 2023RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models9 Jun
  5. 2023TextMesh: Generation of Realistic 3D Meshes From Text Prompts24 Apr
  6. 2023Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction13 Apr
  7. 2023Vox-E: Text-guided Voxel Editing of 3D Objects21 Mar
  8. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  9. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation14 Mar
  10. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  11. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 60

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2023Segment Anythingcited by 8,234
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2022Generative Adversarial Networkscited by 5,627
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  12. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
38 more
  1. 2013Auto-Encoding Variational Bayescited by 4,053
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  5. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  6. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  7. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  8. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  9. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  10. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  11. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  12. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
  13. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  14. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  15. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
  16. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  17. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  18. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 927
  19. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  20. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 727
  21. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 642
  22. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  23. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 484
  24. 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 416
  25. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 345
  26. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  27. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  28. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  29. 2023P+: Extended Textual Conditioning in Text-to-Image Generationcited by 242
  30. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  31. 2023HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Modelscited by 225
  32. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 212
  33. 2023StyleDrop: Text-to-Image Generation in Any Stylecited by 199
  34. 2023Teaching CLIP to Count to Tencited by 198
  35. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  36. 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
  37. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  38. 2023DreamEditor: Text-Driven 3D Scene Editing with Neural Fieldscited by 155

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 14; 2026: 12.

38 papers.

Sort:

By area:

  1. 2024RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion modelscited by 161
  2. 2024DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generationcited by 93
  3. 2024TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusioncited by 52
  4. 2024A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesiscited by 37
  5. 2024ZeST: Zero-Shot Material Transfer from a Single Imagecited by 24
  6. 2024Collaborative Control for Geometry-Conditioned PBR Image Generationcited by 22
  7. 2024BlenderAlchemy: Editing 3D Graphics with Vision-Language Modelscited by 22
  8. 2025IntrinsiX: High-Quality PBR Generation using Image Priorscited by 19
  9. 2025Kontinuous Kontext: Continuous Strength Control for Instruction-based Image Editingcited by 18
  10. 2025Generative AI for Cel-Animation: A Surveycited by 17
  11. 2025IntrinsicEdit: Precise generative image manipulation in intrinsic spacecited by 17
  12. 2024ZeroComp: Zero-shot Object Compositing from Image Intrinsics via Diffusioncited by 16
26 more
  1. 2024Uni-Renderer: Unifying Rendering and Inverse Rendering Via Dual Stream Diffusioncited by 15
  2. 2025Materialist: Physically Based Editing Using Single-Image Inverse Renderingcited by 10
  3. 2025MaterialFusion: High-Quality, Zero-Shot, and Controllable Material Transfer with Diffusion Modelscited by 10
  4. 2025MARBLE: Material Recomposition and Blending in CLIP-Spacecited by 9
  5. 2025MirrorVerse: Pushing Diffusion Models to Realistically Reflect the Worldcited by 8
  6. 2025MatSwap: Light-aware material transfers in imagescited by 6
  7. 2024Bootstrap3D: Improving Multi-view Diffusion Model with Synthetic Datacited by 5
  8. 2024LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Modelcited by 5
  9. 2024IntroStyle: Training-Free Introspective Style Attribution using Diffusion Featurescited by 3
  10. 2025ePBR: Extended PBR Materials in Image Synthesiscited by 3
  11. 2025SPIE: Semantic and Structural Post-Training of Image Editing Diffusion Models with AI feedbackcited by 2
  12. 2025A Controllable Appearance Representation for Flexible Transfer and Editingcited by 2
  13. 2026SeeClear: Reliable Transparent Object Depth Estimation via Generative Opacificationcited by 2
  14. 2026TexEditor: Structure-Preserving Text-Driven Texture Editingcited by 1
  15. 2026PixelSmile: Toward Fine-Grained Facial Expression Editingcited by 1
  16. 2026ControlLight: Towards Controllable, Consistent, and Generalizable Low-Light Enhancementcited by 1
  17. 2025CoatFusion: Controllable Material Coating in Imagescited by 0
  18. 2025FROMAT: Multiview Material Appearance Transfer via Few-Shot Self-Attention Adaptationcited by 0
  19. 2026Alterbute: Editing Intrinsic Attributes of Objects in Imagescited by 0
  20. 2026Bridging Rendering and Generative Modeling with Monte Carlo Transport Schedulingcited by 0
  21. 2026FMMC: Harnessing the Power of Foundation Models for Accurate Material Classificationcited by 0
  22. 2026DynaVid: Learning to Generate Highly Dynamic Videos using Synthetic Motion Datacited by 0
  23. 2026DealMaTe: Multi-Dimensional Material Transfer via Diffusion Transformercited by 0
  24. 2026Thinking in Boxes: 3D Editing in Real Images Made Easycited by 0
  25. 2026Controllable Texture Tiling with Transformed RoPE-Enhanced Diffusion Modelscited by 0
  26. 2026MatReplace: A Reference-Free, Conditioning-Aligned Benchmark for Material Replacement in Interior Scenescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Intrinsic Image Diffusion for Indoor Single-view Material Estimation
  2. 2024Material Anything: Generating Materials for Any 3D Object via Diffusion
  3. 2025RealMat: Realistic Materials with Diffusion and Reinforcement Learning
  4. 2025Generative Detail Enhancement for Physically Based Materials
  5. 2024MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors
  6. 2023ControlMat: A Controlled Generative Approach to Material Capture
  7. 2024IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination
  8. 2023Material Palette: Extraction of Materials from a Single Image