# CustomNet: Zero-shot Object Customization with Variable-Viewpoints in Text-to-Image Diffusion Models

arXiv: https://arxiv.org/abs/2310.19784 · HTML: https://mlsanity.com/paper/2310.19784/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 32 of its 42 references that are in the map.
- First posted on arXiv 30 Oct 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022)
- [BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing](https://mlsanity.com/paper/2305.14720/) (2023)
- [ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation](https://mlsanity.com/paper/2302.13848/) (2023)
- [AnyDoor: Zero-shot Object-level Image Customization](https://mlsanity.com/paper/2307.09481/) (2023)
- [Novel View Synthesis with Diffusion Models](https://mlsanity.com/paper/2210.04628/) (2022)
- [Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuning](https://mlsanity.com/paper/2307.11410/) (2023)
- [CodeNeRF: Disentangled Neural Radiance Fields for Object Categories](https://mlsanity.com/paper/2109.01750/) (2021)
- [Cones: Concept Neurons in Diffusion Models for Customized Generation](https://mlsanity.com/paper/2303.05125/) (2023)
- [DoveNet: Deep Image Harmonization via Domain Verification](https://mlsanity.com/paper/1911.13239/) (2019)
- [Transformation-Grounded Image Generation Network for Novel 3D View Synthesis](https://mlsanity.com/paper/1703.02921/) (2017)
- [High-Resolution Image Harmonization via Collaborative Dual Transformations](https://mlsanity.com/paper/2109.06671/) (2021)

## Built on by (most cited first, 25 of 25)

- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [VideoBooth: Diffusion-based Video Generation with Image Prompts](https://mlsanity.com/paper/2312.00777/) (2023)
- [Zero-shot Image Editing with Reference Imitation](https://mlsanity.com/paper/2406.07547/) (2024)
- [A Survey on Personalized Content Synthesis with Diffusion Models](https://mlsanity.com/paper/2405.05538/) (2024)
- [Personalized Image Generation with Deep Generative Models: A Decade Survey](https://mlsanity.com/paper/2502.13081/) (2025)
- [Thinking Outside the BBox: Unconstrained Generative Object Compositing](https://mlsanity.com/paper/2409.04559/) (2024)
- [Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation](https://mlsanity.com/paper/2408.14819/) (2024)
- [DIVE: Taming DINO for Subject-Driven Video Editing](https://mlsanity.com/paper/2412.03347/) (2024)
- [ZeroComp: Zero-shot Object Compositing from Image Intrinsics via Diffusion](https://mlsanity.com/paper/2410.08168/) (2024)
- [ObjectMate: A Recurrence Prior for Object Insertion and Subject-Driven Generation](https://mlsanity.com/paper/2412.08645/) (2024)
- [GroundingBooth: Grounding Text-to-Image Customization](https://mlsanity.com/paper/2409.08520/) (2024)
- [Diffusion Models are Geometry Critics: Single Image 3D Editing Using Pre-Trained Diffusion Priors](https://mlsanity.com/paper/2403.11503/) (2024)
- [Multitwine: Multi-Object Compositing with Text and Layout Control](https://mlsanity.com/paper/2502.05165/) (2025)
- [BlenderFusion: 3D-Grounded Visual Editing and Generative Compositing](https://mlsanity.com/paper/2506.17450/) (2025)
- [ObjectCompose: Evaluating Resilience of Vision-Based Models on Object-to-Background Compositional Changes](https://mlsanity.com/paper/2403.04701/) (2024)
- [Harmonizing Visual and Textual Embeddings for Zero-Shot Text-to-Image Customization](https://mlsanity.com/paper/2403.14155/) (2024)
- [MObI: Multimodal Object Inpainting Using Diffusion Models](https://mlsanity.com/paper/2501.03173/) (2025)
- [Personalize Your Gaussian: Consistent 3D Scene Personalization from a Single Image](https://mlsanity.com/paper/2505.14537/) (2025)
- [MVCustom: Multi-View Customized Diffusion via Geometric Latent Rendering and Completion](https://mlsanity.com/paper/2510.13702/) (2025)
- [SparsyFed: Sparse Adaptive Federated Training](https://mlsanity.com/paper/2504.05153/) (2025)
- [Zero-Shot Personalized Camera Motion Control for Image-to-Video Synthesis](https://mlsanity.com/paper/2504.09472/) (2025)
- [DepGAN: Leveraging Depth Maps for Handling Occlusions and Transparency in Image Composition](https://mlsanity.com/paper/2407.11890/) (2024)
- [ComposeAnyone: Controllable Layout-to-Human Generation with Decoupled Multimodal Conditions](https://mlsanity.com/paper/2501.12173/) (2025)
- [Reference-Guided Diffusion Inpainting For Multimodal Counterfactual Generation](https://mlsanity.com/paper/2507.23058/) (2025)
- [Reflection Generation for Composite Image Using Diffusion Model](https://mlsanity.com/paper/2604.02168/) (2026)

## Abstract (arXiv)

Incorporating a customized object into image generation presents an attractive feature in text-to-image generation. However, existing optimization-based and encoder-based methods are hindered by drawbacks such as time-consuming optimization, insufficient identity preservation, and a prevalent copy-pasting effect. To overcome these limitations, we introduce CustomNet, a novel object customization approach that explicitly incorporates 3D novel view synthesis capabilities into the object customization process. This integration facilitates the adjustment of spatial position relationships and viewpoints, yielding diverse outputs while effectively preserving object identity. Moreover, we introduce delicate designs to enable location control and flexible background control through textual descriptions or specific user-defined images, overcoming the limitations of existing 3D novel view synthesis methods. We further leverage a dataset construction pipeline that can better handle real-world objects and complex backgrounds. Equipped with these designs, our method facilitates zero-shot object customization without test-time optimization, offering simultaneous control over the viewpoints, location, and background. As a result, our CustomNet ensures enhanced identity preservation and generates diverse, harmonious outputs.