# StructuredMesh: 3D Structured Optimization of Faccade Components on Photogrammetric Mesh Models using Binary Integer Programming

arXiv: https://arxiv.org/abs/2306.04184 · HTML: https://mlsanity.com/paper/2306.04184/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 64 references that are in the map.
- First posted on arXiv 7 Jun 2023.

## Builds on (most cited first, 12 of 12)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Differentiable Ranks and Sorting using Optimal Transport](https://mlsanity.com/paper/1905.11885/) (2019)
- [OpenStreetMap: Challenges and Opportunities in Machine Learning and Remote Sensing](https://mlsanity.com/paper/2007.06277/) (2020)
- [Reconstructing Compact Building Models from Point Clouds Using Deep Implicit Fields](https://mlsanity.com/paper/2112.13142/) (2021)
- [A review of mean-shift algorithms for clustering](https://mlsanity.com/paper/1503.00687/) (2015)
- [Efficient 2D and 3D Facade Segmentation using Auto-Context](https://mlsanity.com/paper/1606.06437/) (2016)
- [Leveraging Photogrammetric Mesh Models for Aerial-Ground Feature Point Matching Toward Integrated 3D Reconstruction](https://mlsanity.com/paper/2002.09085/) (2020)
- [Structure-Aware Completion of Photogrammetric Meshes in Urban Road Environment](https://mlsanity.com/paper/2011.11210/) (2020)
- [Fast and Regularized Reconstruction of Building Faccades from Street-View Images using Binary Integer Programming](https://mlsanity.com/paper/2002.08549/) (2020)
- [Facade Segmentation in the Wild](https://mlsanity.com/paper/1805.08634/) (2018)

## Abstract (arXiv)

The lack of faccade structures in photogrammetric mesh models renders them inadequate for meeting the demands of intricate applications. Moreover, these mesh models exhibit irregular surfaces with considerable geometric noise and texture quality imperfections, making the restoration of structures challenging. To address these shortcomings, we present StructuredMesh, a novel approach for reconstructing faccade structures conforming to the regularity of buildings within photogrammetric mesh models. Our method involves capturing multi-view color and depth images of the building model using a virtual camera and employing a deep learning object detection pipeline to semi-automatically extract the bounding boxes of faccade components such as windows, doors, and balconies from the color image. We then utilize the depth image to remap these boxes into 3D space, generating an initial faccade layout. Leveraging architectural knowledge, we apply binary integer programming (BIP) to optimize the 3D layout's structure, encompassing the positions, orientations, and sizes of all components. The refined layout subsequently informs faccade modeling through instance replacement. We conducted experiments utilizing building mesh models from three distinct datasets, demonstrating the adaptability, robustness, and noise resistance of our proposed methodology. Furthermore, our 3D layout evaluation metrics reveal that the optimized layout enhances precision, recall, and F-score by 6.5%, 4.5%, and 5.5%, respectively, in comparison to the initial layout.