# Mosaic-SDF for 3D Generative Models

arXiv: https://arxiv.org/abs/2312.09222 · HTML: https://mlsanity.com/paper/2312.09222/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 43 of its 52 references that are in the map.
- First posted on arXiv 14 Dec 2023.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Building Normalizing Flows with Stochastic Interpolants](https://mlsanity.com/paper/2209.15571/) (2022)
- [K-Planes: Explicit Radiance Fields in Space, Time, and Appearance](https://mlsanity.com/paper/2301.10241/) (2023)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022)
- [LION: Latent Point Diffusion Models for 3D Shape Generation](https://mlsanity.com/paper/2210.06978/) (2022)
- [3D Shape Generation and Completion through Point-Voxel Diffusion](https://mlsanity.com/paper/2104.03670/) (2021)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models](https://mlsanity.com/paper/2301.11445/) (2023)
- [Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion](https://mlsanity.com/paper/2212.06135/) (2022)
- [Mixture of Volumetric Primitives for Efficient Neural Rendering](https://mlsanity.com/paper/2103.01954/) (2021)
- [3D Neural Field Generation using Triplane Diffusion](https://mlsanity.com/paper/2211.16677/) (2022)
- [Equivariant Flows: Exact Likelihood Generative Learning for Symmetric Densities](https://mlsanity.com/paper/2006.02425/) (2020)

## Built on by (most cited first, 30 of 48)

- [LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation](https://mlsanity.com/paper/2402.05054/) (2024)
- [CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets](https://mlsanity.com/paper/2406.13897/) (2024)
- [Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer](https://mlsanity.com/paper/2405.14832/) (2024)
- [CraftsMan3D: High-fidelity Mesh Generation with 3D Native Generation and Interactive Geometry Refiner](https://mlsanity.com/paper/2405.14979/) (2024)
- [4Diffusion: Multi-view Video Diffusion Model for 4D Generation](https://mlsanity.com/paper/2405.20674/) (2024)
- [3DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusion](https://mlsanity.com/paper/2409.12957/) (2024)
- [LN3DIFF++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generation](https://mlsanity.com/paper/2403.12019/) (2024)
- [V3D: Video Diffusion Models are Effective 3D Generators](https://mlsanity.com/paper/2403.06738/) (2024)
- [EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generation](https://mlsanity.com/paper/2409.18114/) (2024)
- [Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials](https://mlsanity.com/paper/2407.02445/) (2024)
- [PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Models](https://mlsanity.com/paper/2412.18608/) (2024)
- [Meta 3D TextureGen: Fast and Consistent Texture Generation for 3D Objects](https://mlsanity.com/paper/2407.02430/) (2024)
- [Efficient Part-level 3D Object Generation via Dual Volume Packing](https://mlsanity.com/paper/2506.09980/) (2025)
- [SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images](https://mlsanity.com/paper/2501.04689/) (2025)
- [An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion](https://mlsanity.com/paper/2408.03178/) (2024)
- [Gaussian Variation Field Diffusion for High-fidelity Video-to-4D Synthesis](https://mlsanity.com/paper/2507.23785/) (2025)
- [View Selection for 3D Captioning via Diffusion Ranking](https://mlsanity.com/paper/2404.07984/) (2024)
- [FlowSDF: Flow Matching for Medical Image Segmentation Using Distance Transforms](https://mlsanity.com/paper/2405.18087/) (2024)
- [TetSphere Splatting: Representing High-Quality Geometry with Lagrangian Volumetric Meshes](https://mlsanity.com/paper/2405.20283/) (2024)
- [Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representation](https://mlsanity.com/paper/2409.03718/) (2024)
- [LaGeM: A Large Geometry Model for 3D Representation Learning and Diffusion](https://mlsanity.com/paper/2410.01295/) (2024)
- [NeuSDFusion: A Spatial-Aware Generative Model for 3D Shape Completion, Reconstruction, and Generation](https://mlsanity.com/paper/2403.18241/) (2024)
- [A Survey on 3D Human Avatar Modeling -- From Reconstruction to Generation](https://mlsanity.com/paper/2406.04253/) (2024)
- [FlexiDreamer: Single Image-to-3D Generation with FlexiCubes](https://mlsanity.com/paper/2404.00987/) (2024)
- [GaussianAnything: Interactive Point Cloud Flow Matching For 3D Object Generation](https://mlsanity.com/paper/2411.08033/) (2024)
- [GaussianStego: A Generalizable Stenography Pipeline for Generative 3D Gaussians Splatting](https://mlsanity.com/paper/2407.01301/) (2024)
- [Coverage Axis++: Efficient Inner Point Selection for 3D Shape Skeletonization](https://mlsanity.com/paper/2401.12946/) (2024)
- [Wavelet Latent Diffusion (Wala): Billion-Parameter 3D Generative Model with Compact Wavelet Encodings](https://mlsanity.com/paper/2411.08017/) (2024)
- [3D Shape Tokenization via Latent Flow Matching](https://mlsanity.com/paper/2412.15618/) (2024)
- [TetWeave: Isosurface Extraction using On-The-Fly Delaunay Tetrahedral Grids for Gradient-Based Mesh Optimization](https://mlsanity.com/paper/2505.04590/) (2025)

## Abstract (arXiv)

Current diffusion or flow-based generative models for 3D shapes divide to two: distilling pre-trained 2D image diffusion models, and training directly on 3D shapes. When training a diffusion or flow models on 3D shapes a crucial design choice is the shape representation. An effective shape representation needs to adhere three design principles: it should allow an efficient conversion of large 3D datasets to the representation form; it should provide a good tradeoff of approximation power versus number of parameters; and it should have a simple tensorial form that is compatible with existing powerful neural architectures. While standard 3D shape representations such as volumetric grids and point clouds do not adhere to all these principles simultaneously, we advocate in this paper a new representation that does. We introduce Mosaic-SDF (M-SDF): a simple 3D shape representation that approximates the Signed Distance Function (SDF) of a given shape by using a set of local grids spread near the shape's boundary. The M-SDF representation is fast to compute for each shape individually making it readily parallelizable; it is parameter efficient as it only covers the space around the shape's boundary; and it has a simple matrix form, compatible with Transformer-based architectures. We demonstrate the efficacy of the M-SDF representation by using it to train a 3D generative flow model including class-conditioned generation with the 3D Warehouse dataset, and text-to-3D generation using a dataset of about 600k caption-shape pairs.