mlsanityData to 7 Oct 2026

Breathing New Life into 3D Assets with Generative Repainting

arXiv 2309.08523 ↗First posted 15 Sep 2023cs.CV cs.GRcited by 17

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Diffusion-based text-to-image models ignited immense attention from the vision community, artists, and content creators. Broad adoption of these models is due to significant improvement in the quality of generations and efficient conditioning on various modalities, not just text. However, lifting the rich generative priors of these 2D models into 3D is challenging. Recent works have proposed various pipelines powered by the entanglement of diffusion models and neural fields. We explore the power of pretrained 2D diffusion models and standard 3D neural radiance fields as independent, standalone tools and demonstrate their ability to work together in a non-learned fashion. Such modularity has the intrinsic advantage of eased partial upgrades, which became an important property in such a fast-paced domain. Our pipeline accepts any legacy renderable geometry, such as textured or untextured meshes, orchestrates the interaction between 2D generative refinement and 3D consistency enforcement tools, and outputs a painted input geometry in several formats. We conduct a large-scale study on a wide range of objects and categories from the ShapeNetSem dataset and demonstrate the advantages of our approach, both qualitatively and quantitatively. Project page: https://www.obukhov.ai/repainting_3d_assets

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 30 of its 42 references that are in the map. First posted on arXiv 15 Sep 2023.

Builds on

30 of 42 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023TextMesh: Generation of Realistic 3D Meshes From Text Prompts24 Apr
  2. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Models20 Mar
  3. 2023TEXTure: Text-Guided Texturing of 3D Shapes3 Feb
  4. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  5. 2022Novel View Synthesis with Diffusion Models6 Oct
  6. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  7. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Models24 Jan

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  8. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  9. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  10. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  11. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  12. 2018Demystifying MMD GANscited by 1,268
11 more
  1. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  2. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  3. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  4. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  5. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  6. 2022The Role of ImageNet Classes in Fr'echet Inception Distancecited by 211
  7. 20213D-aware Image Synthesis via Learning Structural and Textural Representationscited by 109
  8. 2019Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Researchcited by 98
  9. 2022Pix2NeRF: Unsupervised Conditional π-GAN for Single Image to Neural Radiance Fields Translationcited by 73
  10. 2022Texturify: Generating Textures on 3D Shape Surfacescited by 65
  11. 2022TT-NF: Tensor Train Neural Fieldscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 7; 2025: 3; 2026: 2.

17 papers.

Sort:

By area:

  1. 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 323
  2. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 184
  3. 2024I-Design: Personalized LLM Interior Designercited by 87
  4. 2025Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysiscited by 78
  5. 2023DGInStyle: Domain-Generalizable Semantic Segmentation with Image Diffusion Models and Stylized Semantic Controlcited by 44
  6. 2023Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Modelscited by 29
  7. 2023Repaint123: Fast and High-quality One Image to 3D Generation with Progressive Controllable 2D Repaintingcited by 25
  8. 2024MatAtlas: Text-driven Consistent Geometry Texturing and Material Assignmentcited by 14
  9. 2024RoomTex: Texturing Compositional Indoor Scenes via Iterative Inpaintingcited by 14
  10. 2023Single Mesh Diffusion Models with Field Latents for Texture Generationcited by 11
  11. 2024Flash-Split: 2D Reflection Removal with Flash Cues and Latent Diffusion Separationcited by 10
  12. 2026LaVR: Scene Latent Conditioned Generative Video Trajectory Re-Rendering using Large 4D Reconstruction Modelscited by 10
5 more
  1. 2024VCD-Texture: Variance Alignment based 3D-2D Co-Denoising for Text-Guided Texturingcited by 8
  2. 2025Parametric Shadow Control for Portrait Generation in Text-to-Image Diffusion Modelscited by 5
  3. 2024Consistency^2: Consistent and Fast 3D Painting with Latent Consistency Modelscited by 2
  4. 2024ConTEXTure: Consistent Multiview Images to Texturecited by 0
  5. 2026WaterGen: Decoupling Scene and Medium in Underwater Image Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generation
  2. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion
  3. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior
  4. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation
  5. 2023Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models
  6. 2023Generating Images with 3D Annotations Using Diffusion Models
  7. 2024Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior
  8. 2023Neural Point Cloud Diffusion for Disentangled 3D Shape and Appearance Generation