mlsanityData to 14 Oct 2026

VideoMat: Extracting PBR Materials from Video Diffusion Models

arXiv 2506.09665 ↗First posted 11 Jun 2025cs.GR cs.CVcited by 10

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We leverage finetuned video diffusion models, intrinsic decomposition of videos, and physically-based differentiable rendering to generate high quality materials for 3D models given a text prompt or a single image. We condition a video diffusion model to respect the input geometry and lighting condition. This model produces multiple views of a given 3D model with coherent material properties. Secondly, we use a recent model to extract intrinsics (base color, roughness, metallic) from the generated video. Finally, we use the intrinsics alongside the generated video in a differentiable path tracer to robustly extract PBR materials directly compatible with common content creation tools.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 47 of its 67 references that are in the map. First posted on arXiv 11 Jun 2025.

Builds on

47 of 67 references.

Closest ancestors · 22

Same direction, newest first

  1. 2025DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models30 Jan
  2. 2024MCMat: Multiview-Consistent and Physically Accurate PBR Material Generation18 Dec
  3. 2024CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models27 Nov
  4. 2024TEXGen: a Generative Diffusion Model for Mesh Textures22 Nov
  5. 2024MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors23 Sep
  6. 2024Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video Diffusion Models11 Sep
  7. 2024Neural Gaffer: Relighting Any Object via Diffusion11 Jun
  8. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets30 May
  9. 2024DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models27 May
  10. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Models16 May
  11. 2024RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models1 May
  12. 2024MaPa: Text-driven Photorealistic Material Painting for 3D Shapes26 Apr
10 more
  1. 2024IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination17 Apr
  2. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion18 Mar
  3. 2024DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation19 Feb
  4. 2023Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models21 Dec
  5. 2023HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance30 May
  6. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  7. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation24 Mar
  8. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Models20 Mar
  9. 2023TEXTure: Text-Guided Texturing of 3D Shapes3 Feb
  10. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  7. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  8. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  9. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  10. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  11. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  12. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 770
13 more
  1. 2020Modular Primitives for High-Performance Differentiable Renderingcited by 441
  2. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 366
  3. 2021Extracting Triangular 3D Models, Materials, and Lighting From Imagescited by 366
  4. 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 341
  5. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  6. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 167
  7. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 153
  8. 2023Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Renderingcited by 84
  9. 2024TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusioncited by 52
  10. 2024IllumiNeRF: 3D Relighting Without Inverse Renderingcited by 29
  11. 2025IntrinsiX: High-Quality PBR Generation using Image Priorscited by 19
  12. 2024Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materialscited by 17
  13. 2024MatAtlas: Text-driven Consistent Geometry Texturing and Material Assignmentcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025UniRelight: Learning Joint Decomposition and Synthesis for Video Relightingcited by 40
  2. 2025LuxDiT: Lighting Estimation with Video Diffusion Transformercited by 15
  3. 2026Generative World Renderercited by 10
  4. 2025LaFiTe: A Generative Latent Field for 3D Native Texturingcited by 4
  5. 2026VideoMatGen: PBR Materials through Joint Generative Modelingcited by 3
  6. 2025LumiTex: Towards High-Fidelity PBR Texture Generation with Illumination Contextcited by 2
  7. 2026VideoNeuMat: Neural Material Extraction from Generative Video Modelscited by 2
  8. 2026Advances in Neural 3D Mesh Texturing: A Surveycited by 0
  9. 2026Extracting Neural Materials from Imagescited by 0
  10. 2026Diffusion-Based Material Regularization for Physics-Based Inverse Renderingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Material Palette: Extraction of Materials from a Single Image
  2. 2025SViM3D: Stable Video Material Diffusion for Single Image 3D Generation
  3. 2026Texture Space Material Diffusion
  4. 2025MatPedia: A Universal Generative Foundation for High-Fidelity Material Synthesis
  5. 2025Chord: Chain of Rendering Decomposition for PBR Material Estimation from Generated Texture Images
  6. 2024Boosting 3D Object Generation through PBR Materials
  7. 2025MatE: Material Extraction from Single-Image via Geometric Prior
  8. 2025DualMat: PBR Material Estimation via Coherent Dual-Path Diffusion