# Coin3D: Controllable and Interactive 3D Assets Generation with Proxy-Guided Conditioning

arXiv: https://arxiv.org/abs/2405.08054 · HTML: https://mlsanity.com/paper/2405.08054/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 94 references that are in the map.
- First posted on arXiv 13 May 2024.

## Builds on (most cited first, 30 of 56)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022)
- [Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation](https://mlsanity.com/paper/2303.13873/) (2023)
- [SyncDreamer: Generating Multiview-consistent Images from a Single-view Image](https://mlsanity.com/paper/2309.03453/) (2023)
- [Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation](https://mlsanity.com/paper/2212.00774/) (2022)
- [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://mlsanity.com/paper/2306.16928/) (2023)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model](https://mlsanity.com/paper/2310.15110/) (2023)
- [Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions](https://mlsanity.com/paper/2303.12789/) (2023)
- [StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis](https://mlsanity.com/paper/2110.08985/) (2021)
- [Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis](https://mlsanity.com/paper/2111.04276/) (2021)
- [GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images](https://mlsanity.com/paper/2209.11163/) (2022)
- [Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors](https://mlsanity.com/paper/2306.17843/) (2023)

## Built on by (most cited first, 26 of 26)

- [Rethinking Score Distillation as a Bridge Between Image Distributions](https://mlsanity.com/paper/2406.09417/) (2024)
- [3D Mesh Editing using Masked LRMs](https://mlsanity.com/paper/2412.08641/) (2024)
- [HiScene: Creating Hierarchical 3D Scenes with Isometric View Generation](https://mlsanity.com/paper/2504.13072/) (2025)
- [Kiss3DGen: Repurposing Image Diffusion Models for 3D Asset Generation](https://mlsanity.com/paper/2503.01370/) (2025)
- [CMD: Controllable Multiview Diffusion for 3D Editing and Progressive Generation](https://mlsanity.com/paper/2505.07003/) (2025)
- [PrimitiveAnything: Human-Crafted 3D Primitive Assembly Generation with Auto-Regressive Transformer](https://mlsanity.com/paper/2505.04622/) (2025)
- [InstaScene: Towards Complete 3D Instance Decomposition and Reconstruction from Cluttered Scenes](https://mlsanity.com/paper/2507.08416/) (2025)
- [SpaceControl: Introducing Test-Time Spatial Control to 3D Generative Modeling](https://mlsanity.com/paper/2512.05343/) (2025)
- [FlexGen: Flexible Multi-View Generation from Text and Image Inputs](https://mlsanity.com/paper/2410.10745/) (2024)
- [StyleSculptor: Zero-Shot Style-Controllable 3D Asset Generation with Texture-Geometry Dual Guidance](https://mlsanity.com/paper/2509.13301/) (2025)
- [VideoFrom3D: 3D Scene Video Generation via Complementary Image and Video Diffusion Models](https://mlsanity.com/paper/2509.17985/) (2025)
- [Birth and Death of a Rose](https://mlsanity.com/paper/2412.05278/) (2024)
- [ART-DECO: Arbitrary Text Guidance for 3D Detailizer Construction](https://mlsanity.com/paper/2505.20431/) (2025)
- [PEGAsus: 3D Personalization of Geometry and Appearance](https://mlsanity.com/paper/2602.08198/) (2026)
- [A3D: Does Diffusion Dream about 3D Alignment?](https://mlsanity.com/paper/2406.15020/) (2024)
- [SAT-SKYLINES: 3D Building Generation from Satellite Imagery and Coarse Geometric Priors](https://mlsanity.com/paper/2508.18531/) (2025)
- [AnchorDS: Anchoring Dynamic Sources for Semantically Consistent Text-to-3D Generation](https://mlsanity.com/paper/2511.11692/) (2025)
- [ShellMaker: Language-Guided Exterior Completion under Structural Constraints](https://mlsanity.com/paper/2606.31680/) (2026)
- [ScribbleSense: Generative Scribble-Based Texture Editing with Intent Prediction](https://mlsanity.com/paper/2601.22455/) (2026)
- [ExCellGen: Fast, Controllable, Photorealistic 3D Scene Generation from a Single Real-World Exemplar](https://mlsanity.com/paper/2412.16253/) (2024)
- [GeoTexBuild: 3D Building Model Generation from Map Footprints](https://mlsanity.com/paper/2504.08419/) (2025)
- [CompoSE: Compositional Synthesis and Editing of 3D Shapes via Part-Aware Control](https://mlsanity.com/paper/2605.19350/) (2026)
- [Arbor: Explicit Geometric Conditioning for Controllable 3D Asset Generation](https://mlsanity.com/paper/2606.23514/) (2026)
- [SceneFrom3D: Geometry-Conditioned Outdoor 3D Scene Generation via View Scheduling with Object-Level Control](https://mlsanity.com/paper/2607.04540/) (2026)
- [MultiCube: Compositional 3D Generation With Part-Level Semantic and Spatial Control](https://mlsanity.com/paper/2608.20448/) (2026)
- [DiDE:Direct Injection with Color-Texture DEcoupling for 3D Stylization](https://mlsanity.com/paper/2610.02044/) (2026)

## Abstract (arXiv)

As humans, we aspire to create media content that is both freely willed and readily controlled. Thanks to the prominent development of generative techniques, we now can easily utilize 2D diffusion methods to synthesize images controlled by raw sketch or designated human poses, and even progressively edit/regenerate local regions with masked inpainting. However, similar workflows in 3D modeling tasks are still unavailable due to the lack of controllability and efficiency in 3D generation. In this paper, we present a novel controllable and interactive 3D assets modeling framework, named Coin3D. Coin3D allows users to control the 3D generation using a coarse geometry proxy assembled from basic shapes, and introduces an interactive generation workflow to support seamless local part editing while delivering responsive 3D object previewing within a few seconds. To this end, we develop several techniques, including the 3D adapter that applies volumetric coarse shape control to the diffusion model, proxy-bounded editing strategy for precise part editing, progressive volume cache to support responsive preview, and volume-SDS to ensure consistent mesh reconstruction. Extensive experiments of interactive generation and editing on diverse shape proxies demonstrate that our method achieves superior controllability and flexibility in the 3D assets generation task.