mlsanityData to 7 Oct 2026

Mosaic-SDF for 3D Generative Models

arXiv 2312.09222 ↗First posted 14 Dec 2023cs.CV cs.GRcited by 48

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Current diffusion or flow-based generative models for 3D shapes divide to two: distilling pre-trained 2D image diffusion models, and training directly on 3D shapes. When training a diffusion or flow models on 3D shapes a crucial design choice is the shape representation. An effective shape representation needs to adhere three design principles: it should allow an efficient conversion of large 3D datasets to the representation form; it should provide a good tradeoff of approximation power versus number of parameters; and it should have a simple tensorial form that is compatible with existing powerful neural architectures. While standard 3D shape representations such as volumetric grids and point clouds do not adhere to all these principles simultaneously, we advocate in this paper a new representation that does. We introduce Mosaic-SDF (M-SDF): a simple 3D shape representation that approximates the Signed Distance Function (SDF) of a given shape by using a set of local grids spread near the shape's boundary. The M-SDF representation is fast to compute for each shape individually making it readily parallelizable; it is parameter efficient as it only covers the space around the shape's boundary; and it has a simple matrix form, compatible with Transformer-based architectures. We demonstrate the efficacy of the M-SDF representation by using it to train a 3D generative flow model including class-conditioned generation with the 3D Warehouse dataset, and text-to-3D generation using a dataset of about 600k caption-shape pairs.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 43 of its 52 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

43 of 52 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  2. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generation9 Mar
  3. 20233DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models26 Jan
  4. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  5. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion12 Dec
  6. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusion2 Dec
  7. 20223D Neural Field Generation using Triplane Diffusion30 Nov
  8. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  9. 2022LION: Latent Point Diffusion Models for 3D Shape Generation12 Oct
  10. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  11. 2021Diffusion Probabilistic Models for 3D Point Cloud Generation2 Mar

Other references · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2022Classifier-Free Diffusion Guidancecited by 5,717
  4. 2022Flow Matching for Generative Modelingcited by 5,426
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  7. 2018Neural Ordinary Differential Equationscited by 3,733
  8. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  9. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  10. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  11. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  12. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
20 more
  1. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  2. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  3. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  4. 2023K-Planes: Explicit Radiance Fields in Space, Time, and Appearancecited by 742
  5. 2020Implicit Geometric Regularization for Learning Shapescited by 707
  6. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  7. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  8. 2021Mixture of Volumetric Primitives for Efficient Neural Renderingcited by 277
  9. 2020Equivariant Flows: Exact Likelihood Generative Learning for Symmetric Densitiescited by 248
  10. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  11. 2023MeshDiffusion: Score-based Generative 3D Mesh Modelingcited by 198
  12. 2022Neural Wavelet-domain Diffusion for 3D Shape Generationcited by 132
  13. 2023Equivariant flow matchingcited by 131
  14. 2022Diffusion-SDF: Text-to-Shape via Voxelized Diffusioncited by 128
  15. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 106
  16. 20223DILG: Irregular Latent Grids for 3D Generative Modelingcited by 92
  17. 2022SDF-StyleGAN: Implicit SDF-Based StyleGAN for 3D Shape Generationcited by 92
  18. 2021Universal Approximation of Functions on Setscited by 62
  19. 2018Robust Watertight Manifold Surface Generation Method for ShapeNet Modelscited by 58
  20. 2023On Kinetic Optimal Probability Paths for Generative Modelscited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 33; 2025: 6; 2026: 9.

48 papers.

Sort:

By area:

  1. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 730
  2. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 444
  3. 2024Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformercited by 210
  4. 2024CraftsMan3D: High-fidelity Mesh Generation with 3D Native Generation and Interactive Geometry Refinercited by 120
  5. 20244Diffusion: Multi-view Video Diffusion Model for 4D Generationcited by 104
  6. 20243DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusioncited by 100
  7. 2024LN3DIFF++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generationcited by 91
  8. 2024V3D: Video Diffusion Models are Effective 3D Generatorscited by 88
  9. 2024EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generationcited by 84
  10. 2024Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materialscited by 66
  11. 2024PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Modelscited by 60
  12. 2024Meta 3D TextureGen: Fast and Consistent Texture Generation for 3D Objectscited by 52
36 more
  1. 2025Efficient Part-level 3D Object Generation via Dual Volume Packingcited by 47
  2. 2025SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Imagescited by 41
  3. 2024An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusioncited by 34
  4. 2025Gaussian Variation Field Diffusion for High-fidelity Video-to-4D Synthesiscited by 33
  5. 2024View Selection for 3D Captioning via Diffusion Rankingcited by 31
  6. 2024FlowSDF: Flow Matching for Medical Image Segmentation Using Distance Transformscited by 25
  7. 2024TetSphere Splatting: Representing High-Quality Geometry with Lagrangian Volumetric Meshescited by 20
  8. 2024Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representationcited by 17
  9. 2024LaGeM: A Large Geometry Model for 3D Representation Learning and Diffusioncited by 16
  10. 2024NeuSDFusion: A Spatial-Aware Generative Model for 3D Shape Completion, Reconstruction, and Generationcited by 13
  11. 2024A Survey on 3D Human Avatar Modeling -- From Reconstruction to Generationcited by 13
  12. 2024FlexiDreamer: Single Image-to-3D Generation with FlexiCubescited by 12
  13. 2024GaussianAnything: Interactive Point Cloud Flow Matching For 3D Object Generationcited by 12
  14. 2024GaussianStego: A Generalizable Stenography Pipeline for Generative 3D Gaussians Splattingcited by 11
  15. 2024Coverage Axis++: Efficient Inner Point Selection for 3D Shape Skeletonizationcited by 10
  16. 2024Wavelet Latent Diffusion (Wala): Billion-Parameter 3D Generative Model with Compact Wavelet Encodingscited by 10
  17. 20243D Shape Tokenization via Latent Flow Matchingcited by 10
  18. 2025TetWeave: Isosurface Extraction using On-The-Fly Delaunay Tetrahedral Grids for Gradient-Based Mesh Optimizationcited by 6
  19. 2024CoFie: Learning Compact Neural Surface Representations with Coordinate Fieldscited by 5
  20. 2024MicroDreamer: Efficient 3D Generation in ~20 Seconds by Score-based Iterative Reconstructioncited by 4
  21. 2024Meta 3D Gencited by 3
  22. 2026Sculpt4D: Generating 4D Shapes via Sparse-Attention Diffusion Transformerscited by 3
  23. 2024CrossSDF: 3D Reconstruction of Thin Structures From Cross-Sectionscited by 2
  24. 2026LiTo: Surface Light Field Tokenizationcited by 2
  25. 2024GALA: Geometry-Aware Local Adaptive Grids for Detailed 3D Generationcited by 1
  26. 2025Shape from Semantics: 3D Shape Generation from Multi-View Semanticscited by 1
  27. 2026ReLi3D: Relightable Multi-view 3D Reconstruction with Disentangled Illuminationcited by 1
  28. 2024Haptic Repurposing with GenAIcited by 0
  29. 2024Geometry Distributionscited by 0
  30. 2025NF-SLAM: Effective, Normalizing Flow-supported Neural Field representations for object-level visual SLAM in automotive applicationscited by 0
  31. 2026Fus3D: Decoding Consolidated 3D Geometry from Feed-forward Geometry Transformer Latentscited by 0
  32. 2026Patchwork: A compact representation for 3D polygonal shapescited by 0
  33. 2026PoseShield: Neural Collision Fields for Human Self-Collision Resolutioncited by 0
  34. 2026HoloTetSphere: Unified TetSphere Mesh Reconstruction for Physical Simulationscited by 0
  35. 2026Points as Tori: Fast Pointwise Signed Distance for Point Cloudscited by 0
  36. 2026OrnaStyler: Ornament-Aware Latent Editing for Content-Preserving 3D Stylizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functions
  2. 2023Locally Attentional SDF Diffusion for Controllable 3D Shape Generation
  3. 2022IC3D: Image-Conditioned 3D Diffusion for Shape Generation
  4. 2022Deep Generative Models on 3D Representations: A Survey
  5. 2025Point-Based Shape Representation Generation with a Correspondence-Preserving Diffusion Model
  6. 2024Make-A-Shape: a Ten-Million-scale 3D Shape Model
  7. 2024Topology-Aware Latent Diffusion for 3D Shape Generation
  8. 2025Repurposing 2D Diffusion Models for 3D Shape Completion