# GroundUp: Rapid Sketch-Based 3D City Massing

arXiv: https://arxiv.org/abs/2407.12739 · HTML: https://mlsanity.com/paper/2407.12739/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 95 references that are in the map.
- First posted on arXiv 17 Jul 2024.

## Builds on (most cited first, 30 of 51)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching](https://mlsanity.com/paper/2109.07547/) (2021)
- [Enforcing geometric constraints of virtual normal for depth prediction](https://mlsanity.com/paper/1907.12209/) (2019)
- [Efficiently Identifying Task Groupings for Multi-Task Learning](https://mlsanity.com/paper/2109.04617/) (2021)
- [Learning to Recover 3D Scene Shape from a Single Image](https://mlsanity.com/paper/2012.09365/) (2020)
- [Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches](https://mlsanity.com/paper/1510.05970/) (2015)
- [Locally Attentional SDF Diffusion for Controllable 3D Shape Generation](https://mlsanity.com/paper/2305.04461/) (2023)
- [Capturing, Reconstructing, and Simulating: the UrbanScene3D Dataset](https://mlsanity.com/paper/2107.04286/) (2021)
- [DemoFusion: Democratising High-Resolution Image Generation With No ](https://mlsanity.com/paper/2311.16973/) (2023)
- [TransformerFusion: Monocular RGB Scene Reconstruction using Transformers](https://mlsanity.com/paper/2107.02191/) (2021)
- [Monocular Depth Estimation using Diffusion Models](https://mlsanity.com/paper/2302.14816/) (2023)
- [SimpleRecon: 3D Reconstruction Without 3D Convolutions](https://mlsanity.com/paper/2208.14743/) (2022)
- [DiffusionDepth: Diffusion Denoising Approach for Monocular Depth Estimation](https://mlsanity.com/paper/2303.05021/) (2023)
- [CityDreamer: Compositional Generative Model of Unbounded 3D Cities](https://mlsanity.com/paper/2309.00610/) (2023)
- [Learning Shape Priors for Single-View 3D Completion and Reconstruction](https://mlsanity.com/paper/1809.05068/) (2018)
- [Monocular Depth Estimation Based On Deep Learning: An Overview](https://mlsanity.com/paper/2003.06620/) (2020)
- [DeepVideoMVS: Multi-View Stereo on Video with Recurrent Spatio-Temporal Fusion](https://mlsanity.com/paper/2012.02177/) (2020)
- [InfiniCity: Infinite-Scale City Synthesis](https://mlsanity.com/paper/2301.09637/) (2023)

## Built on by (most cited first, 4 of 4)

- [Computer-Aided Layout Generation for Building Design: A Review](https://mlsanity.com/paper/2504.09694/) (2025)
- [Deep Sketch-Based 3D Modeling: A Survey](https://mlsanity.com/paper/2603.03287/) (2026)
- [SketchingReality: From Freehand Scene Sketches To Photorealistic Images](https://mlsanity.com/paper/2602.14648/) (2026)
- [Sketch2BIM: A Multi-Agent Human-AI Collaborative Pipeline to Convert Hand-Drawn Floor Plans to 3D BIM](https://mlsanity.com/paper/2510.20838/) (2025)

## Abstract (arXiv)

We propose GroundUp, the first sketch-based ideation tool for 3D city massing of urban areas. We focus on early-stage urban design, where sketching is a common tool and the design starts from balancing building volumes (masses) and open spaces. With Human-Centered AI in mind, we aim to help architects quickly revise their ideas by easily switching between 2D sketches and 3D models, allowing for smoother iteration and sharing of ideas. Inspired by feedback from architects and existing workflows, our system takes as a first input a user sketch of multiple buildings in a top-down view. The user then draws a perspective sketch of the envisioned site. Our method is designed to exploit the complementarity of information in the two sketches and allows users to quickly preview and adjust the inferred 3D shapes. Our model has two main components. First, we propose a novel sketch-to-depth prediction network for perspective sketches that exploits top-down sketch shapes. Second, we use depth cues derived from the perspective sketch as a condition to our diffusion model, which ultimately completes the geometry in a top-down view. Thus, our final 3D geometry is represented as a heightfield, allowing users to construct the city `from the ground up'.