mlsanityData to 14 Oct 2026

Material Anything: Generating Materials for Any 3D Object via Diffusion

arXiv 2411.15138 ↗First posted 22 Nov 2024cs.CV cs.GRcited by 31

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We present Material Anything, a fully-automated, unified diffusion framework designed to generate physically-based materials for 3D objects. Unlike existing methods that rely on complex pipelines or case-specific optimizations, Material Anything offers a robust, end-to-end solution adaptable to objects under diverse lighting conditions. Our approach leverages a pre-trained image diffusion model, enhanced with a triple-head architecture and rendering loss to improve stability and material quality. Additionally, we introduce confidence masks as a dynamic switcher within the diffusion model, enabling it to effectively handle both textured and texture-less objects across varying lighting conditions. By employing a progressive material generation strategy guided by these confidence masks, along with a UV-space material refiner, our method ensures consistent, UV-ready material outputs. Extensive experiments demonstrate our approach outperforms existing methods across a wide range of object categories and lighting conditions.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 43 of its 52 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

43 of 52 references.

Closest ancestors · 22

Same direction, newest first

  1. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets30 May
  2. 2024DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models27 May
  3. 2024RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models1 May
  4. 2024MaPa: Text-driven Photorealistic Material Painting for 3D Shapes26 Apr
  5. 20243DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priors4 Mar
  6. 2024DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation19 Feb
  7. 2023Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models21 Dec
  8. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model10 Nov
  9. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior25 Oct
  10. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model23 Oct
  11. 2023HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generation2 Oct
  12. 2023ControlMat: A Controlled Generative Approach to Material Capture4 Sep
10 more
  1. 2023MATLABER: Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR18 Aug
  2. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  3. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior24 Mar
  4. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation24 Mar
  5. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Models20 Mar
  6. 2023TEXTure: Text-Guided Texturing of 3D Shapes3 Feb
  7. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  8. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion12 Dec
  9. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  10. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 21

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2023Segment Anythingcited by 8,234
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  6. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  7. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  8. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  9. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 739
  10. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  11. 2020Modular Primitives for High-Performance Differentiable Renderingcited by 441
  12. 2021Extracting Triangular 3D Models, Materials, and Lighting From Imagescited by 366
9 more
  1. 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 258
  2. 20243DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusioncited by 101
  3. 2023Text-Guided Texturing by Synchronized Multi-View Diffusioncited by 86
  4. 2023Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Renderingcited by 84
  5. 2024ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidancecited by 59
  6. 2024FlashTex: Fast Relightable Mesh Texturing with LightControlNetcited by 56
  7. 2024Collaborative Control for Geometry-Conditioned PBR Image Generationcited by 22
  8. 2024Phidias: A Generative Model for Creating 3D Content from Text, Image, and 3D Conditions with Reference-Augmented Diffusioncited by 19
  9. 2024ThemeStation: Generating Theme-Aware 3D Assets from Few Exemplarscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 20; 2026: 10.

31 papers.

Sort:

By area:

  1. 2025Hunyuan3D 2.5: Towards High-Fidelity 3D Assets Generation with Ultimate Detailscited by 125
  2. 2025RomanTex: Decoupling 3D-aware Rotary Positional Embedded Multi-Attention Network for Texture Synthesiscited by 20
  3. 2024Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusioncited by 19
  4. 2025IntrinsiX: High-Quality PBR Generation using Image Priorscited by 19
  5. 2025LightSwitch: Multi-view Relighting with Material-guided Diffusioncited by 15
  6. 2025MuMA: 3D PBR Texturing via Multi-Channel Multi-View Generation and Agentic Post-Processingcited by 8
  7. 2025Video Perception Models for 3D Scene Synthesiscited by 7
  8. 2025ZeroScene: A Zero-Shot Framework for 3D Scene Generation from a Single Image and Controllable Texture Editingcited by 5
  9. 2025Auto3R: Automated 3D Reconstruction and Scanning via Data-driven Uncertainty Quantificationcited by 4
  10. 2025LaFiTe: A Generative Latent Field for 3D Native Texturingcited by 4
  11. 2025ePBR: Extended PBR Materials in Image Synthesiscited by 3
  12. 2025StableIntrinsic: Detail-preserving One-step Diffusion Model for Multi-view Material Estimationcited by 2
19 more
  1. 2025Large Material Gaussian Model for Relightable 3D Generationcited by 2
  2. 2025NI-Tex: Non-isometric Image-based Garment Texture Generationcited by 2
  3. 2025MatMart: Material Reconstruction of 3D Objects via Diffusioncited by 2
  4. 2025LumiTex: Towards High-Fidelity PBR Texture Generation with Illumination Contextcited by 2
  5. 2025Photo3D: Advancing Photorealistic 3D Generation through Structure-Aligned Detail Enhancementcited by 2
  6. 2026AssetGen: Deployable 3D Asset Generation at Interactive Speedcited by 2
  7. 2025MatLat: Material Latent Space for PBR Texture Generationcited by 1
  8. 2026MV2UV: Generating High-quality UV Texture Maps with Multiview Promptscited by 1
  9. 2026Reflecting Process Expertise in Procedural Material Generationcited by 1
  10. 2025FROMAT: Multiview Material Appearance Transfer via Few-Shot Self-Attention Adaptationcited by 0
  11. 2025XDen-1K: A Density Field Dataset of Real-World Objectscited by 0
  12. 2025An evaluation of SVBRDF Prediction from Generative Image Models for Appearance Modeling of 3D Scenescited by 0
  13. 2026Material Magic Wand: Material-Aware Grouping of 3D Parts in Untextured Meshescited by 0
  14. 2026FMMC: Harnessing the Power of Foundation Models for Accurate Material Classificationcited by 0
  15. 2026Advances in Neural 3D Mesh Texturing: A Surveycited by 0
  16. 2026BRDFusion: Physics Meets Generation for Urban Scene Inverse Renderingcited by 0
  17. 2026Diffusion-Based Material Regularization for Physics-Based Inverse Renderingcited by 0
  18. 2026TaoTex: Boosting Texture Detail Fidelity for Native 3D Material Generationcited by 0
  19. 2026Texture Space Material Diffusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Toward Richer Material Generation via Procedural Data Enhancement
  2. 2024Boosting 3D Object Generation through PBR Materials
  3. 2024MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors
  4. 2024IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination
  5. 2025MaterialFusion: High-Quality, Zero-Shot, and Controllable Material Transfer with Diffusion Models
  6. 2025SViM3D: Stable Video Material Diffusion for Single Image 3D Generation
  7. 2024MaterialPicker: Multi-Modal DiT-Based Material Generation
  8. 2023Alchemist: Parametric Control of Material Properties with Diffusion Models