# Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models

arXiv: https://arxiv.org/abs/2312.13763 · HTML: https://mlsanity.com/paper/2312.13763/

## Key facts

- Cited by 169 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 100 of its 115 references that are in the map.
- First posted on arXiv 21 Dec 2023.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 100)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)

## Built on by (most cited first, 30 of 169)

- [LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation](https://mlsanity.com/paper/2402.05054/) (2024)
- [CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets](https://mlsanity.com/paper/2406.13897/) (2024)
- [GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation](https://mlsanity.com/paper/2403.14621/) (2024)
- [A Survey on 3D Gaussian Splatting](https://mlsanity.com/paper/2401.03890/) (2024)
- [DreamGaussian4D: Generative 4D Gaussian Splatting](https://mlsanity.com/paper/2312.17142/) (2023)
- [PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation](https://mlsanity.com/paper/2404.13026/) (2024)
- [CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models](https://mlsanity.com/paper/2411.18613/) (2024)
- [VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control](https://mlsanity.com/paper/2407.12781/) (2024)
- [WorldScore: A Unified Evaluation Benchmark for World Generation](https://mlsanity.com/paper/2504.00983/) (2025)
- [AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers](https://mlsanity.com/paper/2411.18673/) (2024)
- [OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding](https://mlsanity.com/paper/2406.02058/) (2024)
- [L4GM: Large 4D Gaussian Reconstruction Model](https://mlsanity.com/paper/2406.10324/) (2024)
- [SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency](https://mlsanity.com/paper/2407.17470/) (2024)
- [Recent Advances in 3D Gaussian Splatting](https://mlsanity.com/paper/2403.11134/) (2024)
- [Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesis](https://mlsanity.com/paper/2405.14868/) (2024)
- [3D Gaussian as a New Era: A Survey](https://mlsanity.com/paper/2402.07181/) (2024)
- [4Diffusion: Multi-view Video Diffusion Model for 4D Generation](https://mlsanity.com/paper/2405.20674/) (2024)
- [STAG4D: Spatial-Temporal Anchored Generative 4D Gaussians](https://mlsanity.com/paper/2403.14939/) (2024)
- [GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation](https://mlsanity.com/paper/2403.12365/) (2024)
- [Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attention](https://mlsanity.com/paper/2405.11616/) (2024)
- [TesserAct: Learning 4D Embodied World Models](https://mlsanity.com/paper/2504.20995/) (2025)
- [Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting](https://mlsanity.com/paper/2403.15624/) (2024)
- [Physics3D: Learning Physical Properties of 3D Gaussians via Video Diffusion](https://mlsanity.com/paper/2406.04338/) (2024)
- [3D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunities](https://mlsanity.com/paper/2407.17418/) (2024)
- [TC4D: Trajectory-Conditioned Text-to-4D Generation](https://mlsanity.com/paper/2403.17920/) (2024)
- [ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuning](https://mlsanity.com/paper/2411.05003/) (2024)
- [4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models](https://mlsanity.com/paper/2406.07472/) (2024)
- [DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos](https://mlsanity.com/paper/2405.02280/) (2024)
- [SC4D: Sparse-Controlled Video-to-4D Generation and Motion Transfer](https://mlsanity.com/paper/2404.03736/) (2024)
- [Click-Gaussian: Interactive Segmentation to Any 3D Gaussians](https://mlsanity.com/paper/2407.11793/) (2024)

## Abstract (arXiv)

Text-guided diffusion models have revolutionized image and video generation and have also been successfully used for optimization-based 3D object synthesis. Here, we instead focus on the underexplored text-to-4D setting and synthesize dynamic, animated 3D objects using score distillation methods with an additional temporal dimension. Compared to previous work, we pursue a novel compositional generation-based approach, and combine text-to-image, text-to-video, and 3D-aware multiview diffusion models to provide feedback during 4D object optimization, thereby simultaneously enforcing temporal consistency, high-quality visual appearance and realistic geometry. Our method, called Align Your Gaussians (AYG), leverages dynamic 3D Gaussian Splatting with deformation fields as 4D representation. Crucial to AYG is a novel method to regularize the distribution of the moving 3D Gaussians and thereby stabilize the optimization and induce motion. We also propose a motion amplification mechanism as well as a new autoregressive synthesis scheme to generate and combine multiple 4D sequences for longer generation. These techniques allow us to synthesize vivid dynamic scenes, outperform previous work qualitatively and quantitatively and achieve state-of-the-art text-to-4D performance. Due to the Gaussian 4D representation, different 4D animations can be seamlessly combined, as we demonstrate. AYG opens up promising avenues for animation, simulation and digital content creation as well as synthetic data generation.