mlsanityData to 7 Oct 2026

GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds

arXiv 2104.07659 ↗First posted 15 Apr 2021cs.CVcited by 98

Not on the map. Why?

Abstract (arXiv)

We present GANcraft, an unsupervised neural rendering framework for generating photorealistic images of large 3D block worlds such as those created in Minecraft. Our method takes a semantic block world as input, where each block is assigned a semantic label such as dirt, grass, or water. We represent the world as a continuous volumetric function and train our model to render view-consistent photorealistic images for a user-controlled camera. In the absence of paired ground truth real images for the block world, we devise a training technique based on pseudo-ground truth and adversarial training. This stands in contrast to prior work on neural rendering for view synthesis, which requires ground truth images to estimate scene geometry and view-dependent appearance. In addition to camera trajectory, GANcraft allows user control over both scene semantics and output style. Experimental results with comparison to strong baselines show the effectiveness of GANcraft on this novel task of photorealistic 3D block world synthesis. The project website is available at https://nvlabs.github.io/GANcraft/ .

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Builds on 53 of its 71 references that are in the map. First posted on arXiv 15 Apr 2021.

Builds on

53 of 71 references.

References · 53

  1. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  4. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  5. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  6. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  7. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  8. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  9. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  10. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  11. 2018Demystifying MMD GANscited by 1,268
  12. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
38 more
  1. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  2. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  3. 2020Neural Sparse Voxel Fieldscited by 959
  4. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  5. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 857
  6. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  7. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  8. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 750
  9. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  10. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
  11. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  12. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 624
  13. 2017Unsupervised Image-to-Image Translation Networkscited by 602
  14. 2020NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesiscited by 493
  15. 2020NeRD: Neural Reflectance Decomposition from Image Collectionscited by 446
  16. 2020Space-time Neural Irradiance Fields for Free-Viewpoint Videocited by 414
  17. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 401
  18. 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
  19. 2019Neural Point-Based Graphicscited by 326
  20. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  21. 2017Toward Multimodal Image-to-Image Translationcited by 295
  22. 2020Neural Radiance Flow for 4D View Synthesis and Video Processingcited by 270
  23. 2020Learned Initializations for Optimizing Coordinate-Based Neural Representationscited by 240
  24. 2017Geometric GANcited by 232
  25. 2020Free View Synthesiscited by 231
  26. 2019Few-Shot Unsupervised Image-to-Image Translationcited by 222
  27. 2018A DIRT-T Approach to Unsupervised Domain Adaptationcited by 215
  28. 2020AutoInt: Automatic Integration for Fast Neural Volume Renderingcited by 204
  29. 2020Neural Reflectance Fields for Appearance Acquisitioncited by 196
  30. 2018Escaping Plato's Cave: 3D Shape From Adversarial Renderingcited by 170
  31. 2020BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Imagescited by 166
  32. 2020DeRF: Decomposed Radiance Fieldscited by 163
  33. 2020You Only Need Adversarial Supervision for Semantic Image Synthesiscited by 129
  34. 2019Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesiscited by 108
  35. 2020Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Renderingcited by 101
  36. 2020Object-Centric Neural Scene Renderingcited by 90
  37. 2020STaR: Self-supervised Tracking and Reconstruction of Rigid Objects in Motion with Neural Renderingcited by 76
  38. 2020World-Consistent Video-to-Video Synthesiscited by 69

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 28; 2023: 32; 2024: 17; 2025: 8; 2026: 2.

98 papers.

Sort:

By area:

  1. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  2. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  3. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  4. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 614
  5. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  6. 2021Neural Fields in Visual Computing and Beyondcited by 539
  7. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  8. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  9. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 324
  10. 2021Urban Radiance Fieldscited by 263
  11. 2021GRAM: Generative Radiance Manifolds for 3D-Aware Image Generationcited by 216
  12. 2023WonderJourney: Going from Anywhere to Everywherecited by 157
38 more
  1. 2023SUDS: Scalable Urban Dynamic Scenescited by 151
  2. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 151
  3. 2022EpiGRAF: Rethinking training of 3D GANscited by 108
  4. 2022Generative Neural Articulated Radiance Fieldscited by 101
  5. 2022InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Imagescited by 82
  6. 2023CityDreamer: Compositional Generative Model of Unbounded 3D Citiescited by 75
  7. 2023DiffusionRig: Learning Personalized Priors for Facial Appearance Editingcited by 74
  8. 2023SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collectionscited by 72
  9. 2023InfiniCity: Infinite-Scale City Synthesiscited by 68
  10. 2022Unified Implicit Neural Stylizationcited by 67
  11. 2022Texturify: Generating Textures on 3D Shape Surfacescited by 65
  12. 2021GAN-Supervised Dense Visual Alignmentcited by 61
  13. 2022Generative Multiplane Images: Making a 2D GAN 3D-Awarecited by 57
  14. 2024VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Modelscited by 56
  15. 2022MARLIN: Masked Autoencoder for facial video Representation LearnINgcited by 55
  16. 20223D GAN Inversion for Controllable Portrait Image Animationcited by 53
  17. 2021Neural Point Light Fieldscited by 51
  18. 2022DiffDreamer: Towards Consistent Unsupervised Single-view Scene Extrapolation with Conditional Diffusion Modelscited by 51
  19. 2023Representing Volumetric Videos as Dynamic MLP Mapscited by 51
  20. 2022Signal Processing for Implicit Neural Representationscited by 46
  21. 2023Persistent Nature: A Generative Model of Unbounded 3D Worldscited by 44
  22. 2024Generative Gaussian Splatting for Unbounded 3D City Generationcited by 43
  23. 2022Disentangled3D: Learning a 3D Generative Model with Disentangled Geometry and Appearance from Monocular Imagescited by 39
  24. 2024Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusioncited by 39
  25. 2023ExBluRF: Efficient Radiance Fields for Extreme Motion Blurred Imagescited by 38
  26. 2023PartNeRF: Generating Part-Aware Editable 3D Shapes without 3D Supervisioncited by 36
  27. 2024A survey of synthetic data augmentation methods in computer visioncited by 34
  28. 2022HumanGen: Generating Human Radiance Fields with Explicit Priorscited by 33
  29. 2022DATID-3D: Diversity-Preserved Domain Adaptation Using Text-to-Image Diffusion for 3D Generative Modelcited by 32
  30. 2024Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Modelscited by 32
  31. 2022Playable Environments: Video Manipulation in Space and Timecited by 29
  32. 2022Sem2NeRF: Converting Single-View Semantic Masks to Neural Radiance Fieldscited by 29
  33. 2023Sat2Density: Faithful Density Learning from Satellite-Ground Image Pairscited by 29
  34. 2024Sat2Scene: 3D Urban Scene Generation from Satellite Images with Diffusioncited by 27
  35. 2023UrbanGIRAFFE: Representing Urban Scenes as Compositional Generative Neural Feature Fieldscited by 26
  36. 2024Coin3D: Controllable and Interactive 3D Assets Generation with Proxy-Guided Conditioningcited by 26
  37. 2023GINA-3D: Learning to Generate Implicit Neural Assets in the Wildcited by 25
  38. 2023CityGen: Infinite and Controllable City Layout Generationcited by 22

50 most cited of 98 All 98 (JSON)