mlsanityData to 7 Oct 2026

Vitruvio: 3D Building Meshes via Single Perspective Sketches

arXiv 2210.13634 ↗First posted 24 Oct 2022cs.CV cs.AI cs.GRcited by 5

Not on the map. Why?

Abstract (arXiv)

Today's architectural engineering and construction (AEC) software require a learning curve to generate a three-dimension building representation. This limits the ability to quickly validate the volumetric implications of an initial design idea communicated via a single sketch. Allowing designers to translate a single sketch to a 3D building will enable owners to instantly visualize 3D project information without the cognitive load required. If previous state-of-the-art (SOTA) data-driven methods for single view reconstruction (SVR) showed outstanding results in the reconstruction process from a single image or sketch, they lacked specific applications, analysis, and experiments in the AEC. Therefore, this research addresses this gap, introducing the first deep learning method focused only on buildings that aim to convert a single sketch to a 3D building mesh: Vitruvio. Vitruvio adapts Occupancy Network for SVR tasks on a specific building dataset (Manhattan 1K). This adaptation brings two main improvements. First, it accelerates the inference process by more than 26% (from 0.5s to 0.37s). Second, it increases the reconstruction accuracy (measured by the Chamfer Distance) by 18%. During this adaptation in the AEC domain, we evaluate the effect of the building orientation in the learning procedure since it constitutes an important design factor. While aligning all the buildings to a canonical pose improved the overall quantitative metrics, it did not capture fine-grain details in more complex building shapes (as shown in our qualitative analysis). Finally, Vitruvio outputs a 3D-printable building mesh with arbitrary topology and genus from a single perspective sketch, providing a step forward to allow owners and designers to communicate 3D information via a 2D, effective, intuitive, and universal communication medium: the sketch.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 74 of its 136 references that are in the map. First posted on arXiv 24 Oct 2022.

Builds on

74 of 136 references.

References · 74

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  8. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  9. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  10. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  11. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  12. 2016Variational Inference: A Review for Statisticianscited by 1,466
38 more
  1. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  2. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  3. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  4. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  5. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  6. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  7. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 672
  8. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 649
  9. 2021UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstructioncited by 643
  10. 2018Deep Visual Domain Adaptation: A Surveycited by 634
  11. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  12. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  13. 2021Neural Fields in Visual Computing and Beyondcited by 539
  14. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  15. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  16. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  17. 2016OctNet: Learning Deep 3D Representations at High Resolutionscited by 340
  18. 2018ABC: A Big CAD Model Dataset For Geometric Deep Learningcited by 339
  19. 2019DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstructioncited by 332
  20. 2017A Neural Representation of Sketch Drawingscited by 311
  21. 2021Vector Neurons: A General Framework for SO(3)-Equivariant Networkscited by 304
  22. 2020State of the Art on Neural Renderingcited by 276
  23. 2020PolyGen: An Autoregressive Generative Model of 3D Meshescited by 267
  24. 2019What Do Single-view 3D Reconstruction Networks Learn?cited by 259
  25. 2022SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Imagecited by 165
  26. 2017InfoVAE: Information Maximizing Variational Autoencoderscited by 124
  27. 2019Superquadrics Revisited: Learning 3D Shape Parsing beyond Cuboidscited by 123
  28. 2020Differentiable Rendering: A Surveycited by 123
  29. 2017MarrNet: 3D Shape Reconstruction via 2.5D Sketchescited by 108
  30. 2019Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformationcited by 103
  31. 2019Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Eracited by 87
  32. 2020House-GAN: Relational Generative Adversarial Networks for Graph-constrained House Layout Generationcited by 86
  33. 2022Learning to generate line drawings that convey geometry and semanticscited by 82
  34. 2021Multimodal Conditional Image Synthesis with Product-of-Experts GANscited by 75
  35. 2021Vitruvion: A Generative Model of Parametric CAD Sketchescited by 66
  36. 2019StructureNet: Hierarchical Graph Networks for 3D Shape Generationcited by 59
  37. 2020Deep Learning for Free-Hand Sketch: A Surveycited by 59
  38. 2021Point2Cyl: Reverse Engineering 3D Objects from Point Clouds to Extrusion Cylinderscited by 59

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 0; 2026: 1.

5 papers.

  1. 2024Text2BIM: Generating Building Models Using a Large Language Model-based Multi-Agent Frameworkcited by 9
  2. 2024GroundUp: Rapid Sketch-Based 3D City Massingcited by 4
  3. 2023Multi-task 3D building understanding with multi-modal pretrainingcited by 1
  4. 2023Strokes2Surface: Recovering Curve Networks From 4D Architectural Design Sketchescited by 0
  5. 2026Reconstruction of a 3D wireframe from a single line drawing via generative depth estimationcited by 0