# Interactive 3D Modeling with a Generative Adversarial Network

arXiv: https://arxiv.org/abs/1706.05170 · HTML: https://mlsanity.com/paper/1706.05170/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2017.

## Built on by (most cited first, 18 of 18)

- [Editing Conditional Radiance Fields](https://mlsanity.com/paper/2105.06466/) (2021)
- [Signal Processing for Implicit Neural Representations](https://mlsanity.com/paper/2210.08772/) (2022)
- [Neural Surface Maps](https://mlsanity.com/paper/2103.16942/) (2021)
- [SPAGHETTI: Editing Implicit Shapes Through Part Aware Generation](https://mlsanity.com/paper/2201.13168/) (2022)
- [Self-Supervised Representation Learning for CAD](https://mlsanity.com/paper/2210.10807/) (2022)
- [NeuForm: Adaptive Overfitting for Neural Shape Editing](https://mlsanity.com/paper/2207.08890/) (2022)
- [T2TD: Text-3D Generation Model based on Prior Knowledge Guidance](https://mlsanity.com/paper/2305.15753/) (2023)
- [GEM3D: GEnerative Medial Abstractions for 3D Shape Synthesis](https://mlsanity.com/paper/2402.16994/) (2024)
- [SRIF: Semantic Shape Registration Empowered by Diffusion-based Image Morphing and Flow Estimation](https://mlsanity.com/paper/2409.11682/) (2024)
- [Machine learning for modelling unstructured grid data in computational physics: a review](https://mlsanity.com/paper/2502.09346/) (2025)
- [Neural Convolutional Surfaces](https://mlsanity.com/paper/2204.02289/) (2022)
- [PASTA: Part-Aware Sketch-to-3D Shape Generation with Text-Aligned Prior](https://mlsanity.com/paper/2503.12834/) (2025)
- [Deep Generative Models on 3D Representations: A Survey](https://mlsanity.com/paper/2210.15663/) (2022)
- [3D GANs and Latent Space: A comprehensive survey](https://mlsanity.com/paper/2304.03932/) (2023)
- [Visual design intuition: Predicting dynamic properties of beams from raw cross-section images](https://mlsanity.com/paper/2111.09701/) (2021)
- [Towards AI-Architecture Liberty: A Comprehensive Survey on Design and Generation of Virtual Architecture by Deep Learning](https://mlsanity.com/paper/2305.00510/) (2023)
- [Generating floorplans for various building functionalities via latent diffusion model](https://mlsanity.com/paper/2412.06859/) (2024)
- [ArchComplete: Autoregressive 3D Architectural Design Generation with Hierarchical Diffusion-Based Upsampling](https://mlsanity.com/paper/2412.17957/) (2024)

## Abstract (arXiv)

This paper proposes the idea of using a generative adversarial network (GAN) to assist a novice user in designing real-world shapes with a simple interface. The user edits a voxel grid with a painting interface (like Minecraft). Yet, at any time, he/she can execute a SNAP command, which projects the current voxel grid onto a latent shape manifold with a learned projection operator and then generates a similar, but more realistic, shape using a learned generator network. Then the user can edit the resulting shape and snap again until he/she is satisfied with the result. The main advantage of this approach is that the projection and generation operators assist novice users to create 3D models characteristic of a background distribution of object shapes, but without having to specify all the details. The core new research idea is to use a GAN to support this application. 3D GANs have previously been used for shape generation, interpolation, and completion, but never for interactive modeling. The new challenge for this application is to learn a projection operator that takes an arbitrary 3D voxel model and produces a latent vector on the shape manifold from which a similar and realistic shape can be generated. We develop algorithms for this and other steps of the SNAP processing pipeline and integrate them into a simple modeling tool. Experiments with these algorithms and tool suggest that GANs provide a promising approach to computer-assisted interactive modeling.