mlsanityData to 7 Oct 2026

Collaborative Control for Geometry-Conditioned PBR Image Generation

arXiv 2402.05919 ↗First posted 8 Feb 2024cs.CV cs.GRcited by 22

Not on the map. Why?

Abstract (arXiv)

Graphics pipelines require physically-based rendering (PBR) materials, yet current 3D content generation approaches are built on RGB models. We propose to model the PBR image distribution directly, avoiding photometric inaccuracies in RGB generation and the inherent ambiguity in extracting PBR from RGB. As existing paradigms for cross-modal fine-tuning are not suited for PBR generation due to both a lack of data and the high dimensionality of the output modalities, we propose to train a new PBR model that is tightly linked to a frozen RGB model using a novel cross-network communication paradigm. As the base RGB model is fully frozen, the proposed method retains its general performance and remains compatible with e.g. IPAdapters for that base model.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 78 of its 89 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

78 of 89 references.

References · 78

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2023Segment Anythingcited by 8,158
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  9. 2022Generative Adversarial Networkscited by 5,603
  10. 2016Layer Normalizationcited by 5,579
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  12. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  3. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  4. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  5. 2016Improved Techniques for Training GANscited by 2,938
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  7. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  8. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  9. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  10. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  11. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  12. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  13. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  14. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  15. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  16. 2016Pixel Recurrent Neural Networkscited by 814
  17. 2023Adversarial Diffusion Distillationcited by 741
  18. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  19. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  20. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  21. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Modelcited by 516
  22. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 489
  23. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 308
  24. 2023Rethinking FID: Towards a Better Evaluation Metric for Image Generationcited by 303
  25. 2023LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matchingcited by 290
  26. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 266
  27. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  28. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Modelscited by 234
  29. 2023LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenescited by 228
  30. 2023DreamEditor: Text-Driven 3D Scene Editing with Neural Fieldscited by 154
  31. 2023Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Modelscited by 122
  32. 2023TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Modelscited by 86
  33. 2023Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Renderingcited by 84
  34. 2023DiffusionDepth: Diffusion Denoising Approach for Monocular Depth Estimationcited by 81
  35. 2024Advances in 3D Generation: A Surveycited by 62
  36. 2023Shadows Don't Lie and Lines Can't Bend! Generative Models don't know Projective Geometry...for nowcited by 61
  37. 2024Consistent3D: Towards Consistent High-Fidelity Text-to-3D Generation with Deterministic Sampling Priorcited by 60
  38. 2023Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesiscited by 55

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 13; 2026: 1.

22 papers.

Sort:

By area:

  1. 2025Hunyuan3D 2.5: Towards High-Fidelity 3D Assets Generation with Ultimate Detailscited by 120
  2. 2024SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglementcited by 65
  3. 2024Material Anything: Generating Materials for Any 3D Object via Diffusioncited by 31
  4. 2025MaterialMVP: Illumination-Invariant Material Generation via Multi-view PBR Diffusioncited by 28
  5. 2024GenLit: Reformulating Single-Image Relighting as Video Generationcited by 20
  6. 2025IntrinsiX: High-Quality PBR Generation using Image Priorscited by 18
  7. 2024Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representationcited by 17
  8. 2024Uni-Renderer: Unifying Rendering and Inverse Rendering Via Dual Stream Diffusioncited by 15
  9. 2025TransPixeler: Advancing Text-to-Video Generation with Transparencycited by 12
  10. 2024Boosting 3D Object Generation through PBR Materialscited by 10
  11. 2024ARM: Appearance Reconstruction Model for Relightable 3D Generationcited by 7
  12. 2025SViM3D: Stable Video Material Diffusion for Single Image 3D Generationcited by 7
10 more
  1. 2025Stable Part Diffusion 4D: Multi-View RGB and Kinematic Parts Video Generationcited by 5
  2. 2024Jointly Generating Multi-view Consistent PBR Textures using Collaborative Controlcited by 4
  3. 2025SuperCarver: Texture-Consistent 3D Geometry Super-Resolution for High-Fidelity Surface Detail Generationcited by 2
  4. 2025A Controllable Appearance Representation for Flexible Transfer and Editingcited by 2
  5. 2025LumiTex: Towards High-Fidelity PBR Texture Generation with Illumination Contextcited by 2
  6. 2025DualMat: PBR Material Estimation via Coherent Dual-Path Diffusioncited by 1
  7. 2026ReLi3D: Relightable Multi-view 3D Reconstruction with Disentangled Illuminationcited by 1
  8. 2025LumiX: Structured and Coherent Text-to-Intrinsic Generationcited by 0
  9. 2025FROMAT: Multiview Material Appearance Transfer via Few-Shot Self-Attention Adaptationcited by 0
  10. 2025An evaluation of SVBRDF Prediction from Generative Image Models for Appearance Modeling of 3D Scenescited by 0