# AutoPoly: Predicting a Polygonal Mesh Construction Sequence from a Silhouette Image

arXiv: https://arxiv.org/abs/2203.15233 · HTML: https://mlsanity.com/paper/2203.15233/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 36 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 11 of 11)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [Deep Fashion3D: A Dataset and Benchmark for 3D Garment Reconstruction from Single Images](https://mlsanity.com/paper/2003.12753/) (2020)
- [Learning Shape Priors for Single-View 3D Completion and Reconstruction](https://mlsanity.com/paper/1809.05068/) (2018)
- [Neural Marching Cubes](https://mlsanity.com/paper/2106.11272/) (2021)
- [Fusion 360 Gallery: A Dataset and Environment for Programmatic CAD Construction from Human Design Sequences](https://mlsanity.com/paper/2010.02392/) (2020)
- [Modeling 3D Shapes by Reinforcement Learning](https://mlsanity.com/paper/2003.12397/) (2020)
- [Intuitive and Efficient Roof Modeling for Reconstruction and Synthesis](https://mlsanity.com/paper/2109.07683/) (2021)
- [ClipFlip : Multi-view Clipart Design](https://mlsanity.com/paper/2008.12933/) (2020)

## Abstract (arXiv)

Polygonal modeling is a core task of content creation in Computer Graphics. The complexity of modeling, in terms of the number and the order of operations and time required to execute them makes it challenging to learn and execute. Our goal is to automatically derive a polygonal modeling sequence for a given target. Then, one can learn polygonal modeling by observing the resulting sequence and also expedite the modeling process by starting from the auto-generated result. As a starting point for building a system for 3D modeling in the future, we tackle the 2D shape modeling problem and present AutoPoly, a hybrid method that generates a polygonal mesh construction sequence from a silhouette image. The key idea of our method is the use of the Monte Carlo tree search (MCTS) algorithm and differentiable rendering to separately predict sequential topological actions and geometric actions. Our hybrid method can alter topology, whereas the recently proposed inverse shape estimation methods using differentiable rendering can only handle a fixed topology. Our novel reward function encourages MCTS to select topological actions that lead to a simpler shape without self-intersection. We further designed two deep learning-based methods to improve the expansion and simulation steps in the MCTS search process: an n-step "future action prediction" network (nFAP-Net) to generate candidates for potential topological actions, and a shape warping network (WarpNet) to predict polygonal shapes given the predicted rendered images and topological actions. We demonstrate the efficiency of our method on 2D polygonal shapes of multiple man-made object categories.