# HyperColor: A HyperNetwork Approach for Synthesizing Auto-colored 3D Models for Game Scenes Population

arXiv: https://arxiv.org/abs/2108.01411 · HTML: https://mlsanity.com/paper/2108.01411/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 50 references that are in the map.
- First posted on arXiv 3 Aug 2021.

## Builds on (most cited first, 28 of 28)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [Wasserstein Auto-Encoders](https://mlsanity.com/paper/1711.01558/) (2017)
- [OctNet: Learning Deep 3D Representations at High Resolutions](https://mlsanity.com/paper/1611.05009/) (2016)
- [Learning Gradient Fields for Shape Generation](https://mlsanity.com/paper/2008.06520/) (2020)
- [Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs](https://mlsanity.com/paper/1703.09438/) (2017)
- [Learning a Predictable and Generative Vector Representation for Objects](https://mlsanity.com/paper/1603.08637/) (2016)
- [Procedural Content Generation via Machine Learning (PCGML)](https://mlsanity.com/paper/1702.00539/) (2017)
- [PointGrow: Autoregressively Learned Point Cloud Generation with Self-Attention](https://mlsanity.com/paper/1810.05591/) (2018)
- [Patch-based Progressive 3D Point Set Upsampling](https://mlsanity.com/paper/1811.11286/) (2018)
- [Multiresolution Tree Networks for 3D Point Cloud Processing](https://mlsanity.com/paper/1807.03520/) (2018)
- [Point Cloud GAN](https://mlsanity.com/paper/1810.05795/) (2018)
- [Category-Specific Object Reconstruction from a Single Image](https://mlsanity.com/paper/1411.6069/) (2014)
- [Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers](https://mlsanity.com/paper/1804.10975/) (2018)
- [ColorNet: Investigating the importance of color spaces for image classification](https://mlsanity.com/paper/1902.00267/) (2019)
- [Hypernetwork approach to generating point clouds](https://mlsanity.com/paper/2003.00802/) (2020)
- [Cramer-Wold AutoEncoder](https://mlsanity.com/paper/1805.09235/) (2018)
- [Real-World Textured Things: a Repository of Textured Models Generated with Modern Photo-Reconstruction Tools](https://mlsanity.com/paper/2004.14753/) (2020)
- [A Progressive Conditional Generative Adversarial Network for Generating Dense and Colored 3D Point Clouds](https://mlsanity.com/paper/2010.05391/) (2020)
- [Point Cloud Colorization Based on Densely Annotated 3D Shape Dataset](https://mlsanity.com/paper/1810.05396/) (2018)
- [HyperFlow: Representing 3D Objects as Surfaces](https://mlsanity.com/paper/2006.08710/) (2020)

## Built on by (most cited first, 1 of 1)

- [HyperPlanes: Hypernetwork Approach to Rapid NeRF Adaptation](https://mlsanity.com/paper/2402.01524/) (2024)

## Abstract (arXiv)

Designing a 3D game scene is a tedious task that often requires a substantial amount of work. Typically, this task involves synthesis, coloring, and placement of 3D models within the game scene. To lessen this workload, we can apply machine learning to automate some aspects of the game scene development. Earlier research has already tackled automated generation of the game scene background with machine learning. However, model auto-coloring remains an underexplored problem. The automatic coloring of a 3D model is a challenging task, especially when dealing with the digital representation of a colorful, multipart object. In such a case, we have to ``understand'' the object's composition and coloring scheme of each part. Existing single-stage methods have their own caveats such as the need for segmentation of the object or generating individual parts that have to be assembled together to yield the final model. We address these limitations by proposing a two-stage training approach to synthesize auto-colored 3D models. In the first stage, we obtain a 3D point cloud representing a 3D object, whilst in the second stage, we assign colors to points within such cloud. Next, by leveraging the so-called triangulation trick, we generate a 3D mesh in which the surfaces are colored based on interpolation of colored points representing vertices of a given mesh triangle. This approach allows us to generate a smooth coloring scheme. Experimental evaluation shows that our two-stage approach gives better results in terms of shape reconstruction and coloring when compared to traditional single-stage techniques.