# TwinTex: Geometry-aware Texture Generation for Abstracted 3D Architectural Models

arXiv: https://arxiv.org/abs/2309.11258 · HTML: https://mlsanity.com/paper/2309.11258/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 82 references that are in the map.
- First posted on arXiv 20 Sep 2023.

## Builds on (most cited first, 19 of 19)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures](https://mlsanity.com/paper/2211.07600/) (2022)
- [Capturing, Reconstructing, and Simulating: the UrbanScene3D Dataset](https://mlsanity.com/paper/2107.04286/) (2021)
- [Image inpainting: A review](https://mlsanity.com/paper/1909.06399/) (2019)
- [PyTorch Image Quality: Metrics for Image Quality Assessment](https://mlsanity.com/paper/2208.14818/) (2022)
- [Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting](https://mlsanity.com/paper/1708.01670/) (2017)
- [Single-Perspective Warps in Natural Image Stitching](https://mlsanity.com/paper/1802.04645/) (2018)
- [ELSED: Enhanced Line SEgment Drawing](https://mlsanity.com/paper/2108.03144/) (2021)
- [Quasi-homography warps in image stitching](https://mlsanity.com/paper/1701.08006/) (2017)
- [Plane-Based Optimization of Geometry and Texture for RGB-D Reconstruction of Indoor Scenes](https://mlsanity.com/paper/1905.09829/) (2019)
- [Image Stitching by Line-guided Local Warping with Global Similarity Constraint](https://mlsanity.com/paper/1702.07935/) (2017)
- [Virtual Rephotography: Novel View Prediction Error for 3D Reconstruction](https://mlsanity.com/paper/1601.06950/) (2016)
- [Projective Urban Texturing](https://mlsanity.com/paper/2201.10938/) (2022)

## Built on by (most cited first, 5 of 5)

- [CityGo: Lightweight Urban Modeling and Rendering with Proxy Buildings and Residual Gaussians](https://mlsanity.com/paper/2505.21041/) (2025)
- [SAT-SKYLINES: 3D Building Generation from Satellite Imagery and Coarse Geometric Priors](https://mlsanity.com/paper/2508.18531/) (2025)
- [Aerial Path Planning for Urban Geometry and Texture Co-Capture](https://mlsanity.com/paper/2509.22227/) (2025)
- [CRAYM: Neural Field Optimization via Camera RAY Matching](https://mlsanity.com/paper/2412.01618/) (2024)
- [DiffTex: Differentiable Texturing for Architectural Proxy Models](https://mlsanity.com/paper/2509.23336/) (2025)

## Abstract (arXiv)

Coarse architectural models are often generated at scales ranging from individual buildings to scenes for downstream applications such as Digital Twin City, Metaverse, LODs, etc. Such piece-wise planar models can be abstracted as twins from 3D dense reconstructions. However, these models typically lack realistic texture relative to the real building or scene, making them unsuitable for vivid display or direct reference. In this paper, we present TwinTex, the first automatic texture mapping framework to generate a photo-realistic texture for a piece-wise planar proxy. Our method addresses most challenges occurring in such twin texture generation. Specifically, for each primitive plane, we first select a small set of photos with greedy heuristics considering photometric quality, perspective quality and facade texture completeness. Then, different levels of line features (LoLs) are extracted from the set of selected photos to generate guidance for later steps. With LoLs, we employ optimization algorithms to align texture with geometry from local to global. Finally, we fine-tune a diffusion model with a multi-mask initialization component and a new dataset to inpaint the missing region. Experimental results on many buildings, indoor scenes and man-made objects of varying complexity demonstrate the generalization ability of our algorithm. Our approach surpasses state-of-the-art texture mapping methods in terms of high-fidelity quality and reaches a human-expert production level with much less effort. Project page: https://vcc.tech/research/2023/TwinTex.