# 3DILG: Irregular Latent Grids for 3D Generative Modeling

arXiv: https://arxiv.org/abs/2205.13914 · HTML: https://mlsanity.com/paper/2205.13914/

## Key facts

- Cited by 93 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 72 references that are in the map.
- First posted on arXiv 27 May 2022.

## Builds on (most cited first, 30 of 58)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [PointConv: Deep Convolutional Networks on 3D Point Clouds](https://mlsanity.com/paper/1811.07246/) (2018)
- [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [Masked Autoencoders for Point Cloud Self-supervised Learning](https://mlsanity.com/paper/2203.06604/) (2022)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [Local Implicit Grid Representations for 3D Scenes](https://mlsanity.com/paper/2003.08981/) (2020)

## Built on by (most cited first, 30 of 93)

- [3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models](https://mlsanity.com/paper/2301.11445/) (2023)
- [Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation](https://mlsanity.com/paper/2306.17115/) (2023)
- [3DGen: Triplane Latent Diffusion for Textured Mesh Generation](https://mlsanity.com/paper/2303.05371/) (2023)
- [DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior](https://mlsanity.com/paper/2310.16818/) (2023)
- [Locally Attentional SDF Diffusion for Controllable 3D Shape Generation](https://mlsanity.com/paper/2305.04461/) (2023)
- [CraftsMan3D: High-fidelity Mesh Generation with 3D Native Generation and Interactive Geometry Refiner](https://mlsanity.com/paper/2405.14979/) (2024)
- [SparseFlex: High-Resolution and Arbitrary-Topology 3D Shape Modeling](https://mlsanity.com/paper/2503.21732/) (2025)
- [SALAD: Part-Level Latent Diffusion for 3D Shape Generation and Manipulation](https://mlsanity.com/paper/2303.12236/) (2023)
- [Generative AI meets 3D: A Survey on Text-to-3D in AIGC Era](https://mlsanity.com/paper/2305.06131/) (2023)
- [Mosaic-SDF for 3D Generative Models](https://mlsanity.com/paper/2312.09222/) (2023)
- [OctFusion: Octree-based Diffusion Models for 3D Shape Generation](https://mlsanity.com/paper/2408.14732/) (2024)
- [An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion](https://mlsanity.com/paper/2408.03178/) (2024)
- [OctGPT: Octree-based Multiscale Autoregressive Models for 3D Shape Generation](https://mlsanity.com/paper/2504.09975/) (2025)
- [Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking](https://mlsanity.com/paper/2401.06614/) (2024)
- [Neural Shape Deformation Priors](https://mlsanity.com/paper/2210.05616/) (2022)
- [Make-A-Shape: a Ten-Million-scale 3D Shape Model](https://mlsanity.com/paper/2401.11067/) (2024)
- [MVD2: Efficient Multiview 3D Reconstruction for Multiview Diffusion](https://mlsanity.com/paper/2402.14253/) (2024)
- [ALTO: Alternating Latent Topologies for Implicit 3D Reconstruction](https://mlsanity.com/paper/2212.04096/) (2022)
- [CAD: Photorealistic 3D Generation via Adversarial Distillation](https://mlsanity.com/paper/2312.06663/) (2023)
- [AToM: Amortized Text-to-Mesh using 2D Diffusion](https://mlsanity.com/paper/2402.00867/) (2024)
- [Nautilus: Locality-aware Autoencoder for Scalable Mesh Generation](https://mlsanity.com/paper/2501.14317/) (2025)
- [CLIP-Layout: Style-Consistent Indoor Scene Synthesis with Semantic Furniture Embedding](https://mlsanity.com/paper/2303.03565/) (2023)
- [Neural Vector Fields: Implicit Representation by Explicit Learning](https://mlsanity.com/paper/2303.04341/) (2023)
- [EXIM: A Hybrid Explicit-Implicit Representation for Text-Guided 3D Shape Generation](https://mlsanity.com/paper/2311.01714/) (2023)
- [LaGeM: A Large Geometry Model for 3D Representation Learning and Diffusion](https://mlsanity.com/paper/2410.01295/) (2024)
- [TAR3D: Creating High-Quality 3D Assets via Next-Part Prediction](https://mlsanity.com/paper/2412.16919/) (2024)
- [Atlas Gaussians Diffusion for 3D Generation](https://mlsanity.com/paper/2408.13055/) (2024)
- [Sketch-A-Shape: Zero-Shot Sketch-to-3D Shape Generation](https://mlsanity.com/paper/2307.03869/) (2023)
- [Functional Diffusion](https://mlsanity.com/paper/2311.15435/) (2023)
- [Topology-Aware Latent Diffusion for 3D Shape Generation](https://mlsanity.com/paper/2401.17603/) (2024)

## Abstract (arXiv)

We propose a new representation for encoding 3D shapes as neural fields. The representation is designed to be compatible with the transformer architecture and to benefit both shape reconstruction and shape generation. Existing works on neural fields are grid-based representations with latents defined on a regular grid. In contrast, we define latents on irregular grids, enabling our representation to be sparse and adaptive. In the context of shape reconstruction from point clouds, our shape representation built on irregular grids improves upon grid-based methods in terms of reconstruction accuracy. For shape generation, our representation promotes high-quality shape generation using auto-regressive probabilistic models. We show different applications that improve over the current state of the art. First, we show results for probabilistic shape reconstruction from a single higher resolution image. Second, we train a probabilistic model conditioned on very low resolution images. Third, we apply our model to category-conditioned generation. All probabilistic experiments confirm that we are able to generate detailed and high quality shapes to yield the new state of the art in generative 3D shape modeling.