mlsanityData to 14 Oct 2026

Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models

arXiv 2312.13763 ↗First posted 21 Dec 2023cs.CV cs.LGcited by 169

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Text-guided diffusion models have revolutionized image and video generation and have also been successfully used for optimization-based 3D object synthesis. Here, we instead focus on the underexplored text-to-4D setting and synthesize dynamic, animated 3D objects using score distillation methods with an additional temporal dimension. Compared to previous work, we pursue a novel compositional generation-based approach, and combine text-to-image, text-to-video, and 3D-aware multiview diffusion models to provide feedback during 4D object optimization, thereby simultaneously enforcing temporal consistency, high-quality visual appearance and realistic geometry. Our method, called Align Your Gaussians (AYG), leverages dynamic 3D Gaussian Splatting with deformation fields as 4D representation. Crucial to AYG is a novel method to regularize the distribution of the moving 3D Gaussians and thereby stabilize the optimization and induce motion. We also propose a motion amplification mechanism as well as a new autoregressive synthesis scheme to generate and combine multiple 4D sequences for longer generation. These techniques allow us to synthesize vivid dynamic scenes, outperform previous work qualitatively and quantitatively and achieve state-of-the-art text-to-4D performance. Due to the Gaussian 4D representation, different 4D animations can be seamlessly combined, as we demonstrate. AYG opens up promising avenues for animation, simulation and digital content creation as well as synthetic data generation.

Key facts. Cited by 169 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 100 of its 115 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

100 of 115 references.

Closest ancestors · 29

Same direction, newest first

  1. 2023A Unified Approach for Text- and Image-guided 4D Scene Generation28 Nov
  2. 2023Animate124: Animating One Image to 4D Dynamic Scene24 Nov
  3. 2023WildFusion: Learning 3D-Aware Latent Diffusion Models in View Space22 Nov
  4. 2023MetaDreamer: Efficient Text-to-3D Creation With Disentangling Geometry and Texture16 Nov
  5. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion14 Nov
  6. 2023Text-to-3D with Classifier Score Distillation30 Oct
  7. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior25 Oct
  8. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model23 Oct
  9. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusion23 Oct
  10. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Image7 Sep
  11. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  12. 2023IT3D: Improved Text-to-3D Generation with Explicit View Synthesis22 Aug
17 more
  1. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors30 Jun
  2. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  3. 2023TextMesh: Generation of Realistic 3D Meshes From Text Prompts24 Apr
  4. 2023NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models19 Apr
  5. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation24 Mar
  6. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  7. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  8. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion12 Dec
  9. 2022NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priors6 Dec
  10. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  11. 20223D Neural Field Generation using Triplane Diffusion30 Nov
  12. 2022NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360deg Views29 Nov
  13. 2022TetraDiffusion: Tetrahedral Diffusion Models for 3D Shape Generation23 Nov
  14. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  15. 2022LION: Latent Point Diffusion Models for 3D Shape Generation12 Oct
  16. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  17. 2021Diffusion Probabilistic Models for 3D Point Cloud Generation2 Mar

Other references · 71

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  11. 2022Classifier-Free Diffusion Guidancecited by 5,847
  12. 2016Layer Normalizationcited by 5,669
38 more
  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  4. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  6. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  7. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  8. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  9. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,465
  10. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  11. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,198
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  13. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,157
  14. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  15. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  16. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  17. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 843
  18. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  19. 2023Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstructioncited by 759
  20. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 737
  21. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  22. 2021Score-based Generative Modeling in Latent Spacecited by 636
  23. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  24. 2023HexPlane: A Fast Representation for Dynamic Scenescited by 587
  25. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  26. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  27. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  28. 2020Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Videocited by 471
  29. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 465
  30. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 422
  31. 2023PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamicscited by 377
  32. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 368
  33. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  34. 2023LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matchingcited by 290
  35. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 285
  36. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 279
  37. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  38. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 82; 2025: 59; 2026: 27.

169 papers.

Sort:

By area:

  1. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 739
  2. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 455
  3. 2024GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generationcited by 301
  4. 2024A Survey on 3D Gaussian Splattingcited by 224
  5. 2023DreamGaussian4D: Generative 4D Gaussian Splattingcited by 194
  6. 2024PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generationcited by 191
  7. 2024CAT4D: Create Anything in 4D with Multi-View Video Diffusion Modelscited by 161
  8. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  9. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 156
  10. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 154
  11. 2024OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understandingcited by 145
  12. 2024L4GM: Large 4D Gaussian Reconstruction Modelcited by 144
38 more
  1. 2024SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistencycited by 144
  2. 2024Recent Advances in 3D Gaussian Splattingcited by 131
  3. 2024Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesiscited by 117
  4. 20243D Gaussian as a New Era: A Surveycited by 104
  5. 20244Diffusion: Multi-view Video Diffusion Model for 4D Generationcited by 104
  6. 2024STAG4D: Spatial-Temporal Anchored Generative 4D Gaussianscited by 102
  7. 2024GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creationcited by 100
  8. 2024Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attentioncited by 100
  9. 2025TesserAct: Learning 4D Embodied World Modelscited by 94
  10. 2024Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splattingcited by 82
  11. 2024Physics3D: Learning Physical Properties of 3D Gaussians via Video Diffusioncited by 80
  12. 20243D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunitiescited by 78
  13. 2024TC4D: Trajectory-Conditioned Text-to-4D Generationcited by 77
  14. 2024ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuningcited by 70
  15. 20244Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Modelscited by 64
  16. 2024DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videoscited by 63
  17. 2024SC4D: Sparse-Controlled Video-to-4D Generation and Motion Transfercited by 61
  18. 2024Click-Gaussian: Interactive Segmentation to Any 3D Gaussianscited by 58
  19. 2024FlashSplat: 2D to 3D Gaussian Splatting Segmentation Solved Optimallycited by 58
  20. 2024DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representationcited by 58
  21. 2024Wild-GS: Real-Time Novel View Synthesis from Unconstrained Photo Collectionscited by 56
  22. 2024Animate3D: Animating Any 3D Model with Multi-view Video Diffusioncited by 56
  23. 20244K4DGen: Panoramic 4D Generation at 4K Resolutioncited by 54
  24. 2024Comp4D: LLM-Guided Compositional 4D Scene Generationcited by 50
  25. 2024GenXD: Generating Any 3D and 4D Scenescited by 50
  26. 2025SV4D 2.0: Enhancing Spatio-Temporal Consistency in Multi-View Video Diffusion for High-Quality 4D Generationcited by 48
  27. 2024EG4D: Explicit Generation of 4D Object without Score Distillationcited by 45
  28. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 45
  29. 2025Puppeteer: Rig and Animate Your 3D Modelscited by 43
  30. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 41
  31. 2024Vivid-ZOO: Multi-View Video Generation with Diffusion Modelcited by 40
  32. 2024Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with Dynamic Gaussian Surfelscited by 38
  33. 2024Rethinking Score Distillation as a Bridge Between Image Distributionscited by 38
  34. 2024DreamDrive: Generative 4D Scene Modeling from Street View Imagescited by 35
  35. 2024SplatFlow: Multi-View Rectified Flow Model for 3D Gaussian Splatting Synthesiscited by 34
  36. 2025AnimateAnyMesh: A Feed-Forward 4D Foundation Model for Text-Driven Universal Mesh Animationcited by 32
  37. 2024SAGD: Boundary-Enhanced Segment Anything in 3D Gaussian via Gaussian Decompositioncited by 30
  38. 2024Photorealistic Object Insertion with Diffusion-Guided Inverse Renderingcited by 27

50 most cited of 169 All 169 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Gaussian Variation Field Diffusion for High-fidelity Video-to-4D Synthesis
  2. 20264DGS-Fixer: Generative Sparse-View 4D Gaussian Splatting with Iterative Refinement Guided by Video Diffusion Priors
  3. 2025Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Models
  4. 2025Splat4D: Diffusion-Enhanced 4D Gaussian Splatting for Temporally and Spatially Consistent Content Creation
  5. 2025Generative Gaussian Splatting: Generating 3D Scenes with Video Diffusion Priors
  6. 2025Dynamic-eDiTor: Training-Free Text-Driven 4D Scene Editing with Multimodal Diffusion Transformer
  7. 2025Drag4D: Align Your Motion with Text-Driven 3D Scene Generation
  8. 2024Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models