mlsanityData to 14 Oct 2026

Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation

arXiv 2306.17115 ↗First posted 29 Jun 2023cs.CVcited by 228

Not on the map. Why?

Cited by 36 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a novel alignment-before-generation approach to tackle the challenging task of generating general 3D shapes based on 2D images or texts. Directly learning a conditional generative model from images or texts to 3D shapes is prone to producing inconsistent results with the conditions because 3D shapes have an additional dimension whose distribution significantly differs from that of 2D images and texts. To bridge the domain gap among the three modalities and facilitate multi-modal-conditioned 3D shape generation, we explore representing 3D shapes in a shape-image-text-aligned space. Our framework comprises two models: a Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE) and a conditional Aligned Shape Latent Diffusion Model (ASLDM). The former model encodes the 3D shapes into the shape latent space aligned to the image and text and reconstructs the fine-grained 3D neural fields corresponding to given shape embeddings via the transformer-based decoder. The latter model learns a probabilistic mapping function from the image or text space to the latent shape space. Our extensive experiments demonstrate that our proposed approach can generate higher-quality and more diverse 3D shapes that better semantically conform to the visual or textural conditional inputs, validating the effectiveness of the shape-image-text-aligned space for cross-modality 3D shape generation.

Key facts. Cited by 228 papers in the map (data through 14 Oct 2026). Builds on 65 of its 75 references that are in the map. First posted on arXiv 29 Jun 2023.

Builds on

65 of 75 references.

References · 65

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2016Layer Normalizationcited by 5,669
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  11. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
38 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  2. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  3. 2016Improved Techniques for Training GANscited by 2,970
  4. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  5. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  6. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  7. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  8. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  9. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  10. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  11. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  12. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,205
  13. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  14. 2022Human Motion Diffusion Modelcited by 1,080
  15. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  16. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  17. 2020Convolutional Occupancy Networkscited by 761
  18. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 699
  19. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 588
  20. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 575
  21. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  22. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 553
  23. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  24. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  25. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  26. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  27. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 460
  28. 20233DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Modelscited by 458
  29. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  30. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 409
  31. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 385
  32. 2020Implicit Functions in Feature Space for 3D Shape Reconstruction and Completioncited by 343
  33. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 288
  34. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  35. 2020Learning Gradient Fields for Shape Generationcited by 245
  36. 2022AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generationcited by 208
  37. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  38. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 184

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 39; 2025: 100; 2026: 83.

228 papers.

Sort:

By area:

  1. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 770
  2. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 739
  3. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 455
  4. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 429
  5. 2025Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generationcited by 356
  6. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  7. 2024Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformercited by 210
  8. 2025Native and Compact Structured Latents for 3D Generationcited by 175
  9. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  10. 2025Hunyuan3D 2.1: From Images to High-Fidelity 3D Assets with Production-Ready PBR Materialcited by 165
  11. 2024MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformerscited by 144
  12. 2023State of the Art on Diffusion Models for Visual Computingcited by 127
38 more
  1. 2025Hunyuan3D 2.5: Towards High-Fidelity 3D Assets Generation with Ultimate Detailscited by 125
  2. 2024CraftsMan3D: High-fidelity Mesh Generation with 3D Native Generation and Interactive Geometry Refinercited by 123
  3. 2024MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generationcited by 117
  4. 2025Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attentioncited by 110
  5. 2024MeshAnything V2: Artist-Created Mesh Generation With Adjacent Mesh Tokenizationcited by 104
  6. 2024MeshXL: Neural Coordinate Field for Generative 3D Foundation Modelscited by 103
  7. 20243DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusioncited by 101
  8. 2024Dora: Sampling and Benchmarking for 3D Shape Variational Auto-Encoderscited by 98
  9. 2024EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generationcited by 87
  10. 2025Step1X-3D: Towards High-Fidelity and Controllable Generation of Textured 3D Assetscited by 87
  11. 2025PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformerscited by 84
  12. 2024CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLMcited by 81
  13. 2025SparseFlex: High-Resolution and Arbitrary-Topology 3D Shape Modelingcited by 79
  14. 2024Scaling Mesh Generation via Compressive Tokenizationcited by 73
  15. 2024CAD-Recode: Reverse Engineering CAD Code from Point Cloudscited by 66
  16. 2025DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learningcited by 65
  17. 2025OmniPart: Part-Aware 3D Generation with Semantic Decoupling and Structural Cohesioncited by 61
  18. 2025HoloPart: Generative 3D Part Amodal Segmentationcited by 60
  19. 2024ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidancecited by 59
  20. 2025Efficient Part-level 3D Object Generation via Dual Volume Packingcited by 48
  21. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  22. 2025VoxHammer: Training-Free Precise and Coherent 3D Editing in Native 3D Spacecited by 45
  23. 2025Ultra3D: Efficient and High-Fidelity 3D Generation with Part Attentioncited by 44
  24. 2025X-Part: high fidelity and structure coherent shape decompositioncited by 44
  25. 2025LATTICE: Democratize High-Fidelity 3D Generation at Scalecited by 44
  26. 2025MagicArticulate: Make Your 3D Models Articulation-Readycited by 43
  27. 2025Puppeteer: Rig and Animate Your 3D Modelscited by 43
  28. 2025SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Imagescited by 42
  29. 2024MeshArt: Generating Articulated Meshes with Structure-Guided Transformerscited by 41
  30. 2023ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Modelcited by 39
  31. 2025ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understandingcited by 38
  32. 2025AutoPartGen: Autogressive 3D Part Generation and Discoverycited by 33
  33. 2025Gaussian Variation Field Diffusion for High-fidelity Video-to-4D Synthesiscited by 33
  34. 2025ReconViaGen: Towards Accurate Multi-view 3D Object Reconstruction via Generationcited by 33
  35. 2024Make-Your-3D: Fast and Consistent Subject-Driven 3D Content Generationcited by 32
  36. 2025Anymate: A Dataset and Baselines for Learning 3D Object Riggingcited by 27
  37. 2025Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Surveycited by 27
  38. 2025Unleashing Vecset Diffusion Model for Fast Shape Generationcited by 26

50 most cited of 228 All 228 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Zero3D: Semantic-Driven Multi-Category 3D Shape Generation
  2. 2022IC3D: Image-Conditioned 3D Diffusion for Shape Generation
  3. 2025PASTA: Part-Aware Sketch-to-3D Shape Generation with Text-Aligned Prior
  4. 2024Can Shape-Infused Joint Embeddings Improve Image-Conditioned 3D Diffusion?
  5. 2022ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Model
  6. 2024Topology-Aware Latent Diffusion for 3D Shape Generation
  7. 2026ROAD: Reciprocal-Objective Alignment of Discriminative Semantics for 3D Shape Generation
  8. 2025Evaluating Latent Generative Paradigms for High-Fidelity 3D Shape Completion from a Single Depth Image