mlsanityData to 14 Oct 2026

Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuning

arXiv 2312.13980 ↗First posted 21 Dec 2023cs.CV cs.LGcited by 22

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Multi-view diffusion models, obtained by applying Supervised Finetuning (SFT) to text-to-image diffusion models, have driven recent breakthroughs in text-to-3D research. However, due to the limited size and quality of existing 3D datasets, they still suffer from multi-view inconsistencies and Neural Radiance Field (NeRF) reconstruction artifacts. We argue that multi-view diffusion models can benefit from further Reinforcement Learning Finetuning (RLFT), which allows models to learn from the data generated by themselves and improve beyond their dataset limitations during SFT. To this end, we introduce Carve3D, an improved RLFT algorithm coupled with a novel Multi-view Reconstruction Consistency (MRC) metric, to enhance the consistency of multi-view diffusion models. To measure the MRC metric on a set of multi-view images, we compare them with their corresponding NeRF renderings at the same camera viewpoints. The resulting model, which we denote as Carve3DM, demonstrates superior multi-view consistency and NeRF reconstruction quality than existing models. Our results suggest that pairing SFT with Carve3D's RLFT is essential for developing multi-view-consistent diffusion models, mirroring the standard Large Language Model (LLM) alignment pipeline. Our code, training and testing data, and video results are available at: https://desaixie.github.io/carve-3d.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 37 of its 54 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

37 of 54 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model10 Nov
  2. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model23 Oct
  3. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  4. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization29 Jun
  5. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  6. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  7. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  8. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 29

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  12. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
17 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  3. 2022TensoRF: Tensorial Radiance Fieldscited by 1,280
  4. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  5. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  6. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 843
  7. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  8. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 450
  9. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 448
  10. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  11. 2023Text-to-3D using Gaussian Splattingcited by 299
  12. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 288
  13. 2023LCM-LoRA: A Universal Stable-Diffusion Acceleration Modulecited by 211
  14. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  15. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 173
  16. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 108
  17. 2023Tuning computer vision models with task rewardscited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 14; 2026: 2.

22 papers.

Sort:

By area:

  1. 2024Progressive Autoregressive Video Diffusion Modelscited by 68
  2. 2025Amodal3R: Amodal 3D Reconstruction from Occluded 2D Imagescited by 54
  3. 2024LRM-Zero: Training Large Reconstruction Models with Synthesized Datacited by 35
  4. 2024Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusioncited by 31
  5. 20243D-Adapter: Geometry-Consistent Multi-View Diffusion for High-Quality 3D Generationcited by 19
  6. 2024Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curationcited by 12
  7. 2025Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advancescited by 12
  8. 2025Nabla-R2D3: Effective and Efficient 3D Diffusion Alignment with 2D Rewardscited by 11
  9. 2025DreamCS: Geometry-Aware Text-to-3D Generation with Unpaired 3D Reward Supervisioncited by 7
  10. 2024GECO: Generative Image-to-3D within a SECOndcited by 6
  11. 2025GaMO: Geometry-aware Multi-view Diffusion Outpainting for Sparse-View 3D Reconstructioncited by 4
  12. 2025Voxify3D: Pixel Art Meets Volumetric Renderingcited by 3
10 more
  1. 2025MD-ProjTex: Texturing 3D Shapes with Multi-Diffusion Projectioncited by 2
  2. 2025Refining Few-Step Text-to-Multiview Diffusion via Reinforcement Learningcited by 2
  3. 2025Text2Stereo: Repurposing Stable Diffusion for Stereo Generation with Consistency Rewardscited by 2
  4. 2025AlignCVC: Aligning Cross-View Consistency for Single-Image-to-3D Generationcited by 2
  5. 2025Track, Inpaint, Resplat: Subject-driven 3D and 4D Generation with Progressive Texture Infillingcited by 2
  6. 2026Stroke3D: Lifting 2D strokes into rigged 3D model via latent diffusion modelscited by 2
  7. 2026GeoFlow: Enforcing Implicit Geometric Consistency in Video Generationcited by 2
  8. 2025Kaleidoscopic Background Attack: Disrupting Pose Estimation with Multi-Fold Radial Symmetry Texturescited by 1
  9. 2025RealMat: Realistic Materials with Diffusion and Reinforcement Learningcited by 1
  10. 2025Debiasing Diffusion Priors via 3D Attention for Consistent Gaussian Splattingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Text-Image Conditioned Diffusion for Consistent Text-to-3D Generation
  2. 2023Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction
  3. 2025R-Meshfusion: Reinforcement Learning Powered Sparse-View Mesh Reconstruction with Diffusion Priors
  4. 2023Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views
  5. 2022Novel View Synthesis with Diffusion Models
  6. 2022NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priors
  7. 2024Bootstrap3D: Improving Multi-view Diffusion Model with Synthetic Data
  8. 2024Diffusion Models in 3D Vision: A Survey