# ConRad: Image Constrained Radiance Fields for 3D Generation from a Single Image

arXiv: https://arxiv.org/abs/2311.05230 · HTML: https://mlsanity.com/paper/2311.05230/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 39 references that are in the map.
- First posted on arXiv 9 Nov 2023.

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis](https://mlsanity.com/paper/2104.00677/) (2021)
- [Local Implicit Grid Representations for 3D Scenes](https://mlsanity.com/paper/2003.08981/) (2020)
- [OctNet: Learning Deep 3D Representations at High Resolutions](https://mlsanity.com/paper/1611.05009/) (2016)
- [RealFusion: 360deg Reconstruction of Any Object from a Single Image](https://mlsanity.com/paper/2302.10663/) (2023)
- [EDICT: Exact Diffusion Inversion via Coupled Transformations](https://mlsanity.com/paper/2211.12446/) (2022)
- [Learning Shape Abstractions by Assembling Volumetric Primitives](https://mlsanity.com/paper/1612.00404/) (2016)
- [Pix2Vox: Context-aware 3D Reconstruction from Single and Multi-view Images](https://mlsanity.com/paper/1901.11153/) (2019)
- [NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priors](https://mlsanity.com/paper/2212.03267/) (2022)

## Built on by (most cited first, 5 of 5)

- [LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching](https://mlsanity.com/paper/2311.11284/) (2023)
- [Free3D: Consistent Novel View Synthesis without 3D Representation](https://mlsanity.com/paper/2312.04551/) (2023)
- [GeoDream: Disentangling 2D and Geometric Priors for High-Fidelity and Consistent 3D Generation](https://mlsanity.com/paper/2311.17971/) (2023)
- [BootPIG: Bootstrapping Zero-shot Personalized Image Generation Capabilities in Pretrained Diffusion Models](https://mlsanity.com/paper/2401.13974/) (2024)
- [TV-3DG: Mastering Text-to-3D Customized Generation with Visual Prompt](https://mlsanity.com/paper/2410.21299/) (2024)

## Abstract (arXiv)

We present a novel method for reconstructing 3D objects from a single RGB image. Our method leverages the latest image generation models to infer the hidden 3D structure while remaining faithful to the input image. While existing methods obtain impressive results in generating 3D models from text prompts, they do not provide an easy approach for conditioning on input RGB data. Na"ive extensions of these methods often lead to improper alignment in appearance between the input image and the 3D reconstructions. We address these challenges by introducing Image Constrained Radiance Fields (ConRad), a novel variant of neural radiance fields. ConRad is an efficient 3D representation that explicitly captures the appearance of an input image in one viewpoint. We propose a training algorithm that leverages the single RGB image in conjunction with pretrained Diffusion Models to optimize the parameters of a ConRad representation. Extensive experiments show that ConRad representations can simplify preservation of image details while producing a realistic 3D reconstruction. Compared to existing state-of-the-art baselines, we show that our 3D reconstructions remain more faithful to the input and produce more consistent 3D models while demonstrating significantly improved quantitative performance on a ShapeNet object benchmark.