# Cut-and-Approximate: 3D Shape Reconstruction from Planar Cross-sections with Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2210.12509 · HTML: https://mlsanity.com/paper/2210.12509/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 52 references that are in the map.
- First posted on arXiv 22 Oct 2022.

## Builds on (most cited first, 18 of 18)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution](https://mlsanity.com/paper/1807.03247/) (2018)
- [Learning Shape Abstractions by Assembling Volumetric Primitives](https://mlsanity.com/paper/1612.00404/) (2016)
- [Superquadrics Revisited: Learning 3D Shape Parsing beyond Cuboids](https://mlsanity.com/paper/1904.09970/) (2019)
- [CSGNet: Neural Shape Parser for Constructive Solid Geometry](https://mlsanity.com/paper/1712.08290/) (2017)
- [Synthesizing Programs for Images using Reinforced Adversarial Learning](https://mlsanity.com/paper/1804.01118/) (2018)
- [Learning to Paint With Model-based Deep Reinforcement Learning](https://mlsanity.com/paper/1903.04411/) (2019)
- [Learning Deep Sketch Abstraction](https://mlsanity.com/paper/1804.04804/) (2018)
- [ReAgent: Point Cloud Registration using Imitation and Reinforcement Learning](https://mlsanity.com/paper/2103.15231/) (2021)
- [Unsupervised Doodling and Painting with Improved SPIRAL](https://mlsanity.com/paper/1910.01007/) (2019)
- [Modeling 3D Shapes by Reinforcement Learning](https://mlsanity.com/paper/2003.12397/) (2020)
- [Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrations](https://mlsanity.com/paper/1801.10459/) (2018)
- [Learning to Sketch with Deep Q Networks and Demonstrated Strokes](https://mlsanity.com/paper/1810.05977/) (2018)
- [RLSS: A Deep Reinforcement Learning Algorithm for Sequential Scene Generation](https://mlsanity.com/paper/2206.02544/) (2022)

## Built on by (most cited first, 2 of 2)

- [Slice3D: Multi-Slice, Occlusion-Revealing, Single View 3D Reconstruction](https://mlsanity.com/paper/2312.02221/) (2023)
- [OReX: Object Reconstruction from Planar Cross-sections Using Neural Fields](https://mlsanity.com/paper/2211.12886/) (2022)

## Abstract (arXiv)

Current methods for 3D object reconstruction from a set of planar cross-sections still struggle to capture detailed topology or require a considerable number of cross-sections. In this paper, we present, to the best of our knowledge the first 3D shape reconstruction network to solve this task which additionally uses orthographic projections of the shape. Our method is based on applying a Reinforcement Learning algorithm to learn how to effectively parse the shape using a trial-and-error scheme relying on scalar rewards. This method cuts a part of a 3D shape in each step which is then approximated as a polygon mesh. The agent aims to maximize the reward that depends on the accuracy of surface reconstruction for the approximated parts. We also consider pre-training of the network for faster learning using demonstrations generated by a heuristic approach. Experiments show that our training algorithm which benefits from both imitation learning and also self exploration, learns efficient policies faster, which results the agent to produce visually compelling results.