# CityCraft: A Real Crafter for 3D City Generation

arXiv: https://arxiv.org/abs/2406.04983 · HTML: https://mlsanity.com/paper/2406.04983/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 82 references that are in the map.
- First posted on arXiv 7 Jun 2024.

## Builds on (most cited first, 30 of 57)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [LION: Latent Point Diffusion Models for 3D Shape Generation](https://mlsanity.com/paper/2210.06978/) (2022)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics](https://mlsanity.com/paper/2011.09127/) (2020)
- [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023)
- [VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2303.08320/) (2023)
- [GeoChat: Grounded Large Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2311.15826/) (2023)
- [SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning](https://mlsanity.com/paper/2307.06135/) (2023)
- [Holodeck: Language Guided Generation of 3D Embodied AI Environments](https://mlsanity.com/paper/2312.09067/) (2023)
- [Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models](https://mlsanity.com/paper/2303.11989/) (2023)

## Built on by (most cited first, 26 of 26)

- [AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark](https://mlsanity.com/paper/2410.03051/) (2024)
- [3D and 4D World Modeling: A Survey](https://mlsanity.com/paper/2509.07996/) (2025)
- [PAD: Personalized Alignment of LLMs at Decoding-Time](https://mlsanity.com/paper/2410.04070/) (2024)
- [UrbanWorld: An Urban World Model for 3D City Generation](https://mlsanity.com/paper/2407.11965/) (2024)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability](https://mlsanity.com/paper/2506.13558/) (2025)
- [Compositional Generative Model of Unbounded 4D Cities](https://mlsanity.com/paper/2501.08983/) (2025)
- [Sat2City: 3D City Generation from A Single Satellite Image with Cascaded Latent Diffusion](https://mlsanity.com/paper/2507.04403/) (2025)
- [Simulating the Real World: A Unified Survey of Multimodal Generative Models](https://mlsanity.com/paper/2503.04641/) (2025)
- [Yo'City: Personalized and Boundless 3D Realistic City Scene Generation via Self-Critic Expansion](https://mlsanity.com/paper/2511.18734/) (2025)
- [WorldClaw: Agentic 3D Open-World Generation at Scale](https://mlsanity.com/paper/2608.05248/) (2026)
- [Ego3DT: Tracking Every 3D Object in Ego-centric Videos](https://mlsanity.com/paper/2410.08530/) (2024)
- [Sat2RealCity: Geometry-Aware and Appearance-Controllable 3D Urban Generation from Satellite Imagery](https://mlsanity.com/paper/2511.11470/) (2025)
- [Proc-GS: Procedural Building Generation for City Assembly with 3D Gaussians](https://mlsanity.com/paper/2412.07660/) (2024)
- [EarthCrafter: Scalable 3D Earth Generation via Dual-Sparse Latent Diffusion](https://mlsanity.com/paper/2507.16535/) (2025)
- [MajutsuCity: Language-driven Aesthetic-adaptive City Generation with Controllable 3D Assets and Layouts](https://mlsanity.com/paper/2511.20415/) (2025)
- [DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models](https://mlsanity.com/paper/2503.04240/) (2025)
- [LatticeWorld: A Multimodal Large Language Model-Empowered Framework for Interactive Complex World Generation](https://mlsanity.com/paper/2509.05263/) (2025)
- [Imagine a City: CityGenAgent for Procedural 3D City Generation](https://mlsanity.com/paper/2602.05362/) (2026)
- [From 2D to 3D Cognition: A Brief Survey of General World Models](https://mlsanity.com/paper/2506.20134/) (2025)
- [UrbanWorld2.0: A Multimodal Agentic Framework for Reality-Aligned 3D World Generation at City-Scale](https://mlsanity.com/paper/2511.18005/) (2025)
- [LychSim: A Controllable and Interactive Simulation Framework for Vision Research](https://mlsanity.com/paper/2605.12449/) (2026)
- [GardenDesigner: Encoding Aesthetic Principles into Jiangnan Garden Construction via a Chain of Agents](https://mlsanity.com/paper/2604.01777/) (2026)
- [To See a World in a Living Context: Unified Indoor-Outdoor Urban World Generation](https://mlsanity.com/paper/2608.05879/) (2026)
- [PolyLayout: Hierarchical VLM-Guided Layout Generation Beyond Rectangular Rooms](https://mlsanity.com/paper/2608.10838/) (2026)
- [CityCode: Building Agent-Operable 3D Cities from Sparse Aerial Observations](https://mlsanity.com/paper/2610.18342/) (2026)

## Abstract (arXiv)

City scene generation has gained significant attention in autonomous driving, smart city development, and traffic simulation. It helps enhance infrastructure planning and monitoring solutions. Existing methods have employed a two-stage process involving city layout generation, typically using Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), or Transformers, followed by neural rendering. These techniques often exhibit limited diversity and noticeable artifacts in the rendered city scenes. The rendered scenes lack variety, resembling the training images, resulting in monotonous styles. Additionally, these methods lack planning capabilities, leading to less realistic generated scenes. In this paper, we introduce CityCraft, an innovative framework designed to enhance both the diversity and quality of urban scene generation. Our approach integrates three key stages: initially, a diffusion transformer (DiT) model is deployed to generate diverse and controllable 2D city layouts. Subsequently, a Large Language Model(LLM) is utilized to strategically make land-use plans within these layouts based on user prompts and language guidelines. Based on the generated layout and city plan, we utilize the asset retrieval module and Blender for precise asset placement and scene construction. Furthermore, we contribute two new datasets to the field: 1)CityCraft-OSM dataset including 2D semantic layouts of urban areas, corresponding satellite images, and detailed annotations. 2) CityCraft-Buildings dataset, featuring thousands of diverse, high-quality 3D building assets. CityCraft achieves state-of-the-art performance in generating realistic 3D cities.