# BIM Hyperreality: Data Synthesis Using BIM and Hyperrealistic Rendering for Deep Learning

arXiv: https://arxiv.org/abs/2105.04103 · HTML: https://mlsanity.com/paper/2105.04103/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 30 references that are in the map.
- First posted on arXiv 10 May 2021.

## Builds on (most cited first, 14 of 14)

- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1811.12833/) (2018)
- [Learning from Synthetic Humans](https://mlsanity.com/paper/1701.01370/) (2017)
- [NIPS 2016 Tutorial: Generative Adversarial Networks](https://mlsanity.com/paper/1701.00160/) (2016)
- [Pros and Cons of GAN Evaluation Measures](https://mlsanity.com/paper/1802.03446/) (2018)
- [The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation](https://mlsanity.com/paper/1611.09326/) (2016)
- [Synthesizing Training Data for Object Detection in Indoor Scenes](https://mlsanity.com/paper/1702.07836/) (2017)
- [What Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation?](https://mlsanity.com/paper/1801.06397/) (2018)
- [Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks](https://mlsanity.com/paper/1612.07429/) (2016)
- [Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Training](https://mlsanity.com/paper/1810.07911/) (2018)

## Built on by (most cited first, 1 of 1)

- [Deep Learning from Parametrically Generated Virtual Buildings for Real-World Object Recognition](https://mlsanity.com/paper/2302.05283/) (2023)

## Abstract (arXiv)

Deep learning is expected to offer new opportunities and a new paradigm for the field of architecture. One such opportunity is teaching neural networks to visually understand architectural elements from the built environment. However, the availability of large training datasets is one of the biggest limitations of neural networks. Also, the vast majority of training data for visual recognition tasks is annotated by humans. In order to resolve this bottleneck, we present a concept of a hybrid system using both building information modeling (BIM) and hyperrealistic (photorealistic) rendering to synthesize datasets for training a neural network for building object recognition in photos. For generating our training dataset BIMrAI, we used an existing BIM model and a corresponding photo-realistically rendered model of the same building. We created methods for using renderings to train a deep learning model, trained a generative adversarial network (GAN) model using these methods, and tested the output model on real-world photos. For the specific case study presented in this paper, our results show that a neural network trained with synthetic data; i.e., photorealistic renderings and BIM-based semantic labels, can be used to identify building objects from photos without using photos in the training data. Future work can enhance the presented methods using available BIM models and renderings for more generalized mapping and description of photographed built environments.