mlsanityData to 7 Oct 2026

YouDream: Generating Anatomically Controllable Consistent Text-to-3D Animals

arXiv 2406.16273 ↗First posted 24 Jun 2024cs.CVcited by 0

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

3D generation guided by text-to-image diffusion models enables the creation of visually compelling assets. However previous methods explore generation based on image or text. The boundaries of creativity are limited by what can be expressed through words or the images that can be sourced. We present YouDream, a method to generate high-quality anatomically controllable animals. YouDream is guided using a text-to-image diffusion model controlled by 2D views of a 3D pose prior. Our method generates 3D animals that are not possible to create using previous text-to-3D generative methods. Additionally, our method is capable of preserving anatomic consistency in the generated animals, an area where prior text-to-3D approaches often struggle. Moreover, we design a fully automated pipeline for generating commonly found animals. To circumvent the need for human intervention to create a 3D pose, we propose a multi-agent LLM that adapts poses from a limited library of animal 3D poses to represent the desired animal. A user study conducted on the outcomes of YouDream demonstrates the preference of the animal models generated by our method over others. Turntable results and code are released at https://youdream3d.github.io/

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 56 of its 62 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

56 of 62 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation2 Dec
  2. 2023Control3D: Towards Controllable Text-to-3D Generation9 Nov
  3. 2023Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Models4 Oct
  4. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  5. 2023ARTIC3D: Learning Robust Articulated 3D Shapes from Noisy Web Image Collections7 Jun
  6. 2023HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance30 May
  7. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  8. 2023Farm3D: Learning Articulated 3D Animals by Distilling 2D Diffusion20 Apr
  9. 2023DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Models3 Apr
  10. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation24 Mar
  11. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  12. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation14 Mar
2 more
  1. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  2. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  8. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  9. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  10. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  11. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  12. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
30 more
  1. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  2. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 459
  3. 2022Visual Classification via Description from Large Language Modelscited by 331
  4. 20163D Menagerie: Modeling the 3D shape and pose of animalscited by 310
  5. 2023LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matchingcited by 290
  6. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  7. 2023MeshGPT: Generating Triangle Meshes with Decoder-Only Transformerscited by 254
  8. 2023MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusioncited by 210
  9. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  10. 2023Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyondcited by 139
  11. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Codecited by 125
  12. 2022MagicPony: Learning Articulated 3D Animals in the Wildcited by 111
  13. 2022Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understandingcited by 102
  14. 2023DreamHuman: Animatable 3D Avatars from Textcited by 102
  15. 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 91
  16. 2023DreamWaltz: Make a Scene with Complex 3D Animatable Avatarscited by 82
  17. 2023ChatPose: Chatting about 3D Human Posecited by 71
  18. 2022LASSIE: Learning Articulated Shapes from Sparse Image Ensemble via 3D Part Discoverycited by 65
  19. 2021DOVE: Learning Deformable 3D Objects by Watching Videoscited by 61
  20. 2023AvatarVerse: High-quality & Stable 3D Avatar Creation from Text and Posecited by 60
  21. 2022AvatarGen: A 3D Generative Model for Animatable Human Avatarscited by 59
  22. 2024Learning the 3D Fauna of the Webcited by 58
  23. 2024Improved Zero-Shot Classification by Adapting VLMs with Text Descriptionscited by 52
  24. 2023Animal3D: A Comprehensive Dataset of 3D Animal Pose and Shapecited by 43
  25. 20203D Bird Reconstruction: a Dataset, Model, and Shape Recovery from a Single Viewcited by 40
  26. 2023ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Modelcited by 38
  27. 2022Hi-LASSIE: High-Fidelity Articulated Shape and Skeleton Discovery from Sparse Image Ensemblecited by 30
  28. 20243D-PreMise: Can Large Language Models Generate 3D Shapes with Sharp Features and Parametric Control?cited by 21
  29. 2021A Novel Dataset for Keypoint Detection of quadruped Animals from Imagescited by 18
  30. 2023MVControl: Adding Conditional Control to Multi-view Diffusion for Controllable Text-to-3D Generationcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior
  2. 2025How to Move Your Dragon: Text-to-Motion Synthesis for Large-Vocabulary Objects
  3. 2023PODIA-3D: Domain Adaptation of 3D Generative Model Across Large Domain Gap Using Pose-Preserved Text-to-Image Diffusion
  4. 2024DreamBeast: Distilling 3D Fantastical Animals with Part-Aware Knowledge Transfer
  5. 2024Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation
  6. 2023ControlDreamer: Blending Geometry and Style in Text-to-3D
  7. 2024MUSES: 3D-Controllable Image Generation via Multi-Modal Agent Collaboration
  8. 2024SketchDream: Sketch-based Text-to-3D Generation and Editing