# Make-It-4D: Synthesizing a Consistent Long-Term Dynamic Scene Video from a Single Image

arXiv: https://arxiv.org/abs/2308.10257 · HTML: https://mlsanity.com/paper/2308.10257/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 41 references that are in the map.
- First posted on arXiv 20 Aug 2023.

## Builds on (most cited first, 30 of 35)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2303.08320/) (2023)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [Motion Representations for Articulated Animation](https://mlsanity.com/paper/2104.11280/) (2021)
- [Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image](https://mlsanity.com/paper/2012.09855/) (2020)
- [3D Photography using Context-aware Layered Depth Inpainting](https://mlsanity.com/paper/2004.04727/) (2020)
- [Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesis](https://mlsanity.com/paper/1909.12224/) (2019)
- [SceneScape: Text-Driven Consistent Scene Generation](https://mlsanity.com/paper/2302.01133/) (2023)
- [Pathdreamer: A World Model for Indoor Navigation](https://mlsanity.com/paper/2105.08756/) (2021)
- [Deep Image Spatial Transformation for Person Image Generation](https://mlsanity.com/paper/2003.00696/) (2020)
- [PixelSynth: Generating a 3D-Consistent Experience from a Single Image](https://mlsanity.com/paper/2108.05892/) (2021)
- [InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Images](https://mlsanity.com/paper/2207.11148/) (2022)
- [SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections](https://mlsanity.com/paper/2302.01330/) (2023)
- [Aligning Latent and Image Spaces to Connect the Unconnectable](https://mlsanity.com/paper/2104.06954/) (2021)
- [Flow-Grounded Spatial-Temporal Video Prediction from Still Images](https://mlsanity.com/paper/1807.09755/) (2018)
- [Animating Pictures with Eulerian Motion Fields](https://mlsanity.com/paper/2011.15128/) (2020)
- [Look Outside the Room: Synthesizing A Consistent Long-Term 3D Scene Video from A Single Image](https://mlsanity.com/paper/2203.09457/) (2022)
- [Controllable Animation of Fluid Elements in Still Images](https://mlsanity.com/paper/2112.03051/) (2021)
- [Persistent Nature: A Generative Model of Unbounded 3D Worlds](https://mlsanity.com/paper/2303.13515/) (2023)

## Built on by (most cited first, 19 of 19)

- [State of the Art on Diffusion Models for Visual Computing](https://mlsanity.com/paper/2310.07204/) (2023)
- [I-Design: Personalized LLM Interior Designer](https://mlsanity.com/paper/2404.02838/) (2024)
- [Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion](https://mlsanity.com/paper/2407.13759/) (2024)
- [3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation](https://mlsanity.com/paper/2403.09439/) (2024)
- [Hierarchical Patch Diffusion Models for High-Resolution Video Generation](https://mlsanity.com/paper/2406.07792/) (2024)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [DreamMover: Leveraging the Prior of Diffusion Models for Image Interpolation with Large Motion](https://mlsanity.com/paper/2409.09605/) (2024)
- [ElastoGen: 4D Generative Elastodynamics](https://mlsanity.com/paper/2405.15056/) (2024)
- [Towards Robust Monocular Depth Estimation in Non-Lambertian Surfaces](https://mlsanity.com/paper/2408.06083/) (2024)
- [DoF-Gaussian: Controllable Depth-of-Field for 3D Gaussian Splatting](https://mlsanity.com/paper/2503.00746/) (2025)
- [GausSim: Foreseeing Reality by Gaussian Simulator for Elastic Objects](https://mlsanity.com/paper/2412.17804/) (2024)
- [Optimizing 4D Gaussians for Dynamic Scene Video from Single Landscape Images](https://mlsanity.com/paper/2504.05458/) (2025)
- [SoMA: A Real-to-Sim Neural Simulator for Robotic Soft-body Manipulation](https://mlsanity.com/paper/2602.02402/) (2026)
- [S-DyRF: Reference-Based Stylized Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2403.06205/) (2024)
- [Learning an Implicit Physics Model for Image-based Fluid Simulation](https://mlsanity.com/paper/2508.08254/) (2025)
- [SM4Depth: Seamless Monocular Metric Depth Estimation across Multiple Cameras and Scenes by One Model](https://mlsanity.com/paper/2403.08556/) (2024)
- [Pixel-to-4D: Camera-Controlled Image-to-Video Generation with Dynamic 3D Gaussians](https://mlsanity.com/paper/2601.00678/) (2026)
- [Dehallu3D: Hallucination-Mitigated 3D Generation from Single Image via Cyclic View Consistency Refinement](https://mlsanity.com/paper/2603.01601/) (2026)
- [LivingWorld: Interactive 4D World Generation with Environmental Dynamics](https://mlsanity.com/paper/2604.01641/) (2026)

## Abstract (arXiv)

We study the problem of synthesizing a long-term dynamic video from only a single image. This is challenging since it requires consistent visual content movements given large camera motions. Existing methods either hallucinate inconsistent perpetual views or struggle with long camera trajectories. To address these issues, it is essential to estimate the underlying 4D (including 3D geometry and scene motion) and fill in the occluded regions. To this end, we present Make-It-4D, a novel method that can generate a consistent long-term dynamic video from a single image. On the one hand, we utilize layered depth images (LDIs) to represent a scene, and they are then unprojected to form a feature point cloud. To animate the visual content, the feature point cloud is displaced based on the scene flow derived from motion estimation and the corresponding camera pose. Such 4D representation enables our method to maintain the global consistency of the generated dynamic video. On the other hand, we fill in the occluded regions by using a pretrained diffusion model to inpaint and outpaint the input image. This enables our method to work under large camera motions. Benefiting from our design, our method can be training-free which saves a significant amount of training time. Experimental results demonstrate the effectiveness of our approach, which showcases compelling rendering results.