mlsanityData to 7 Oct 2026

VideoMatGen: PBR Materials through Joint Generative Modeling

arXiv 2603.16566 ↗First posted 17 Mar 2026cs.CV cs.GRcited by 3

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We present a method for generating physically-based materials for 3D shapes based on a video diffusion transformer architecture. Our method is conditioned on input geometry and a text description, and jointly models multiple material properties (base color, roughness, metallicity, height map) to form physically plausible materials. We further introduce a custom variational auto-encoder which encodes multiple material modalities into a compact latent space, which enables joint generation of multiple modalities without increasing the number of tokens. Our pipeline generates high-quality materials for 3D shapes given a text prompt, compatible with common content creation tools.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 59 of its 74 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

59 of 74 references.

Closest ancestors · 23

Same direction, newest first

  1. 2025SViM3D: Stable Video Material Diffusion for Single Image 3D Generation9 Oct
  2. 2025VideoMat: Extracting PBR Materials from Video Diffusion Models11 Jun
  3. 2025RomanTex: Decoupling 3D-aware Rotary Positional Embedded Multi-Attention Network for Texture Synthesis24 Mar
  4. 2024TEXGen: a Generative Diffusion Model for Mesh Textures22 Nov
  5. 2024MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors23 Sep
  6. 2024Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video Diffusion Models11 Sep
  7. 2024Neural Gaffer: Relighting Any Object via Diffusion11 Jun
  8. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets30 May
  9. 2024DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models27 May
  10. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Models16 May
  11. 2024RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models1 May
  12. 2024MaPa: Text-driven Photorealistic Material Painting for 3D Shapes26 Apr
11 more
  1. 2024IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination17 Apr
  2. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion18 Mar
  3. 2024DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation19 Feb
  4. 2023Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models21 Dec
  5. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model10 Nov
  6. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  7. 2023HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance30 May
  8. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  9. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Models20 Mar
  10. 2023TEXTure: Text-Guided Texturing of 3D Shapes3 Feb
  11. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 36

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  7. 2022Scalable Diffusion Models with Transformerscited by 5,931
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  11. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  12. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
24 more
  1. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  2. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  3. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  4. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  5. 2020Modular Primitives for High-Performance Differentiable Renderingcited by 439
  6. 2021Extracting Triangular 3D Models, Materials, and Lighting From Imagescited by 363
  7. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 360
  8. 2024GS-LRM: Large Reconstruction Model for 3D Gaussian Splattingcited by 344
  9. 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 325
  10. 2023Rethinking FID: Towards a Better Evaluation Metric for Image Generationcited by 303
  11. 2022The Role of ImageNet Classes in Fr'echet Inception Distancecited by 211
  12. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 162
  13. 2025Native and Compact Structured Latents for 3D Generationcited by 157
  14. 2025VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Modelscited by 112
  15. 2023Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Renderingcited by 84
  16. 2024SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglementcited by 65
  17. 2024FlashTex: Fast Relightable Mesh Texturing with LightControlNetcited by 56
  18. 2024TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusioncited by 51
  19. 2024MatSynth: A Modern PBR Materials Datasetcited by 42
  20. 2025MaterialMVP: Illumination-Invariant Material Generation via Multi-view PBR Diffusioncited by 28
  21. 2023Random-Access Neural Compression of Material Texturescited by 23
  22. 2024Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materialscited by 17
  23. 2025Pandora3D: A Comprehensive Framework for High-Quality 3D Shape and Texture Generationcited by 14
  24. 2025MVPainter: Accurate and Detailed 3D Texture Generation via Multi-View Diffusion with Geometric Controlcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026AssetGen: Deployable 3D Asset Generation at Interactive Speedcited by 2
  2. 2026Extracting Neural Materials from Imagescited by 0
  3. 2026Texture Space Material Diffusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MatPedia: A Universal Generative Foundation for High-Fidelity Material Synthesis
  2. 2024MaterialPicker: Multi-Modal DiT-Based Material Generation
  3. 2026Toward Richer Material Generation via Procedural Data Enhancement
  4. 2024MCMat: Multiview-Consistent and Physically Accurate PBR Material Generation
  5. 2026VideoNeuMat: Neural Material Extraction from Generative Video Models
  6. 2023MATLABER: Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR
  7. 2025MuMA: 3D PBR Texturing via Multi-Channel Multi-View Generation and Agentic Post-Processing
  8. 2024DreamPBR: Text-driven Generation of High-resolution SVBRDF with Multi-modal Guidance