mlsanityData to 14 Oct 2026

Outdoor Scene Extrapolation with Hierarchical Generative Cellular Automata

arXiv 2406.08292 ↗First posted 12 Jun 2024cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

We aim to generate fine-grained 3D geometry from large-scale sparse LiDAR scans, abundantly captured by autonomous vehicles (AV). Contrary to prior work on AV scene completion, we aim to extrapolate fine geometry from unlabeled and beyond spatial limits of LiDAR scans, taking a step towards generating realistic, high-resolution simulation-ready 3D street environments. We propose hierarchical Generative Cellular Automata (hGCA), a spatially scalable conditional 3D generative model, which grows geometry recursively with local kernels following, in a coarse-to-fine manner, equipped with a light-weight planner to induce global consistency. Experiments on synthetic scenes show that hGCA generates plausible scene geometry with higher fidelity and completeness compared to state-of-the-art baselines. Our model generalizes strongly from sim-to-real, qualitatively outperforming baselines on the Waymo-open dataset. We also show anecdotal evidence of the ability to create novel objects from real-world geometric cues even when trained on limited synthetic content. More results and details can be found on https://research.nvidia.com/labs/toronto-ai/hGCA/.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 48 of its 63 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

48 of 63 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  6. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  7. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  8. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  9. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  10. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  11. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  12. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
36 more
  1. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  2. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  3. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  4. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  5. 2020Convolutional Occupancy Networkscited by 761
  6. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 748
  7. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  8. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 553
  9. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  10. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 488
  11. 2016Semantic Scene Completion from a Single Depth Imagecited by 472
  12. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  13. 2018PCN: Point Completion Networkcited by 384
  14. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 380
  15. 2020Local Implicit Grid Representations for 3D Scenescited by 373
  16. 2020Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstructioncited by 321
  17. 2020Learning Gradient Fields for Shape Generationcited by 245
  18. 2020Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completioncited by 223
  19. 2016Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesiscited by 219
  20. 2018Escaping Plato's Cave: 3D Shape From Adversarial Renderingcited by 170
  21. 2020LiDARsim: Realistic LiDAR Simulation by Leveraging the Real Worldcited by 137
  22. 2020SemanticPOSS: A Point Cloud Dataset with Large Quantity of Dynamic Instancescited by 123
  23. 2017ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scanscited by 103
  24. 2020S3CNet: A Sparse Semantic Scene Completion Network for LiDAR Point Cloudscited by 102
  25. 2022Learning to Generate Realistic LiDAR Point Cloudscited by 93
  26. 2023SCPNet: Semantic Scene Completion on Point Cloudcited by 80
  27. 2023UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generationcited by 78
  28. 2019SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scanscited by 77
  29. 2022TorchSparse: Efficient Point Cloud Inference Enginecited by 73
  30. 2020Multimodal Shape Completion via Conditional Generative Adversarial Networkscited by 61
  31. 2019Deep Level Sets: Implicit Surface Representations for 3D Shape Inferencecited by 53
  32. 2020Weakly-supervised 3D Shape Completion in the Wildcited by 33
  33. 2021Learning to Generate 3D Shapes with Generative Cellular Automatacited by 22
  34. 2017Learning to Generate Samples from Noise through Infusion Trainingcited by 15
  35. 2022Probabilistic Implicit Scene Completioncited by 8
  36. 2021Multimodal Shape Completion via IMLEcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2026Seen2Scene: Completing Realistic 3D Scenes with Visibility-Guided Flowcited by 1
  2. 2025Mixtures of Neural Cellular Automata: A Stochastic Framework for Growth Modelling and Self-Organizationcited by 0