mlsanityData to 7 Oct 2026

CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets

arXiv 2406.13897 ↗First posted 30 May 2024cs.CVcited by 444

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

In the realm of digital creativity, our potential to craft intricate 3D worlds from imagination is often hampered by the limitations of existing digital tools, which demand extensive expertise and efforts. To narrow this disparity, we introduce CLAY, a 3D geometry and material generator designed to effortlessly transform human imagination into intricate 3D digital structures. CLAY supports classic text or image inputs as well as 3D-aware controls from diverse primitives (multi-view images, voxels, bounding boxes, point clouds, implicit representations, etc). At its core is a large-scale generative model composed of a multi-resolution Variational Autoencoder (VAE) and a minimalistic latent Diffusion Transformer (DiT), to extract rich 3D priors directly from a diverse range of 3D geometries. Specifically, it adopts neural fields to represent continuous and complete surfaces and uses a geometry generative module with pure transformer blocks in latent space. We present a progressive training scheme to train CLAY on an ultra large 3D model dataset obtained through a carefully designed processing pipeline, resulting in a 3D native geometry generator with 1.5 billion parameters. For appearance generation, CLAY sets out to produce physically-based rendering (PBR) textures by employing a multi-view material diffusion model that can generate 2K resolution textures with diffuse, roughness, and metallic modalities. We demonstrate using CLAY for a range of controllable 3D asset creations, from sketchy conceptual designs to production ready assets with intricate details. Even first time users can easily use CLAY to bring their vivid 3D imaginations to life, unleashing unlimited creativity.

Key facts. Cited by 444 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 72 of its 98 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

72 of 98 references.

Closest ancestors · 26

Same direction, newest first

  1. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models21 Dec
  2. 2023Mosaic-SDF for 3D Generative Models14 Dec
  3. 2023UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generation14 Dec
  4. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D28 Nov
  5. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion14 Nov
  6. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior25 Oct
  7. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model23 Oct
  8. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusion23 Oct
  9. 2023HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusion12 Oct
  10. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Image7 Sep
  11. 2023HD-Fusion: Detailed Text-to-3D Generation Leveraging Multiple Noise Estimation30 Jul
  12. 2023Points-to-3D: Bridging the Gap between Sparse Points and Shape-Controllable Text-to-3D Generation26 Jul
14 more
  1. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors30 Jun
  2. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization29 Jun
  3. 2023DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation21 Jun
  4. 2023HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance30 May
  5. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  6. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation24 Mar
  7. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Models20 Mar
  8. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  9. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generation9 Mar
  10. 2023TEXTure: Text-Guided Texturing of 3D Shapes3 Feb
  11. 20233DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models26 Jan
  12. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  13. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  14. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  10. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  11. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  12. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
34 more
  1. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  2. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  3. 2021Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Datacited by 961
  4. 2020On Layer Normalization in the Transformer Architecturecited by 906
  5. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  6. 2020Convolutional Occupancy Networkscited by 760
  7. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  8. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  9. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  10. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  11. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 459
  12. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  13. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 324
  14. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 308
  15. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 287
  16. 2020PolyGen: An Autoregressive Generative Model of 3D Meshescited by 267
  17. 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 257
  18. 2023MeshGPT: Generating Triangle Meshes with Decoder-Only Transformerscited by 254
  19. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 225
  20. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Modelcited by 213
  21. 2023Text-To-4D Dynamic Scene Generationcited by 204
  22. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 199
  23. 2022SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Viewscited by 175
  24. 2023Locally Attentional SDF Diffusion for Controllable 3D Shape Generationcited by 129
  25. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  26. 2023DreamFace: Progressive Generation of Animatable 3D Faces under Text Guidancecited by 79
  27. 2022Dual Octree Graph Networks for Learning Adaptive Volumetric Shape Representationscited by 71
  28. 2021SA-ConvONet: Sign-Agnostic Optimization of Convolutional Occupancy Networkscited by 59
  29. 2018Robust Watertight Manifold Surface Generation Method for ShapeNet Modelscited by 58
  30. 2023Instant-3D: Instant Neural Radiance Field Training Towards On-Device AR/VR 3D Reconstructioncited by 49
  31. 2020ManifoldPlus: A Robust and Scalable Watertight Manifold Surface Generation Method for Triangle Soupscited by 38
  32. 2019A Skeleton-bridged Deep Learning Approach for Generating Meshes of Complex Topologies from Single RGB Imagescited by 37
  33. 2023Globally Consistent Normal Orientation for Point Clouds by Regularizing the Winding-Number Fieldcited by 37
  34. 2020SkeletonNet: A Topology-Preserving Solution for Learning Mesh Reconstruction of Object Surfaces from RGB Imagescited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 55; 2025: 216; 2026: 173.

444 papers.

Sort:

By area:

  1. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  2. 2025Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generationcited by 342
  3. 2025Hunyuan3D 2.1: From Images to High-Fidelity 3D Assets with Production-Ready PBR Materialcited by 157
  4. 2025Native and Compact Structured Latents for 3D Generationcited by 157
  5. 2025Hunyuan3D 2.5: Towards High-Fidelity 3D Assets Generation with Ultimate Detailscited by 120
  6. 2024MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generationcited by 113
  7. 2024RaDe-GS: Rasterizing Depth in Gaussian Splattingcited by 110
  8. 2025Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attentioncited by 103
  9. 2024MeshAnything V2: Artist-Created Mesh Generation With Adjacent Mesh Tokenizationcited by 102
  10. 2025CAST: Component-Aligned 3D Scene Reconstruction from an RGB Imagecited by 101
  11. 20243DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusioncited by 100
  12. 2025Hi3DGen: High-fidelity 3D Geometry Generation from Images via Normal Bridgingcited by 95
38 more
  1. 2024Dora: Sampling and Benchmarking for 3D Shape Variational Auto-Encoderscited by 93
  2. 2025Step1X-3D: Towards High-Fidelity and Controllable Generation of Textured 3D Assetscited by 85
  3. 2024EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generationcited by 84
  4. 2025Sparc3D: Sparse Representation and Construction for High-Resolution 3D Shapes Modelingcited by 84
  5. 2025PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformerscited by 78
  6. 2025SparseFlex: High-Resolution and Arbitrary-Topology 3D Shape Modelingcited by 76
  7. 2024You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scalecited by 65
  8. 2025DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learningcited by 62
  9. 2025OmniPart: Part-Aware 3D Generation with Semantic Decoupling and Structural Cohesioncited by 60
  10. 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AIcited by 60
  11. 2025Bolt3D: Generating 3D Scenes in Secondscited by 57
  12. 2025HoloPart: Generative 3D Part Amodal Segmentationcited by 57
  13. 2024InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Modelscited by 53
  14. 2025Amodal3R: Amodal 3D Reconstruction from Occluded 2D Imagescited by 52
  15. 2025DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generationcited by 49
  16. 2025SceneGen: Single-Image 3D Scene Generation in One Feedforward Passcited by 49
  17. 2025Efficient Part-level 3D Object Generation via Dual Volume Packingcited by 47
  18. 2025MagicArticulate: Make Your 3D Models Articulation-Readycited by 43
  19. 2025Ultra3D: Efficient and High-Fidelity 3D Generation with Part Attentioncited by 43
  20. 2025X-Part: high fidelity and structure coherent shape decompositioncited by 42
  21. 2025TexVerse: A Universe of 3D Objects with High-Resolution Texturescited by 41
  22. 2025LATTICE: Democratize High-Fidelity 3D Generation at Scalecited by 41
  23. 2025VoxHammer: Training-Free Precise and Coherent 3D Editing in Native 3D Spacecited by 40
  24. 2025PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generationcited by 39
  25. 2024SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAEcited by 38
  26. 2025TreeMeshGPT: Artistic Mesh Generation with Autoregressive Tree Sequencingcited by 38
  27. 2024An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusioncited by 34
  28. 2024MVPaint: Synchronized Multi-View Diffusion for Painting Anything 3Dcited by 34
  29. 2025ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understandingcited by 34
  30. 2025AutoPartGen: Autogressive 3D Part Generation and Discoverycited by 33
  31. 2025Gaussian Variation Field Diffusion for High-fidelity Video-to-4D Synthesiscited by 33
  32. 2025Matrix-3D: Omnidirectional Explorable 3D World Generationcited by 33
  33. 2025EmbodiedGen: Towards a Generative 3D World Engine for Embodied Intelligencecited by 32
  34. 2024Material Anything: Generating Materials for Any 3D Object via Diffusioncited by 31
  35. 2024Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNetscited by 30
  36. 2025SynCity: Training-Free Generation of 3D Worldscited by 30
  37. 2025ReconViaGen: Towards Accurate Multi-view 3D Object Reconstruction via Generationcited by 30
  38. 2025HoLa: B-Rep Generation using a Holistic Latent Representationcited by 29

50 most cited of 444 All 444 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Breathing New Life into 3D Assets with Generative Repainting
  2. 2024Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer
  3. 20243DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priors
  4. 2024Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior
  5. 2022Deep Generative Models on 3D Representations: A Survey
  6. 2026Realiz3D: 3D Generation Made Photorealistic via Domain-Aware Learning
  7. 2024DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data
  8. 2024Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting