# VideoFrom3D: 3D Scene Video Generation via Complementary Image and Video Diffusion Models

arXiv: https://arxiv.org/abs/2509.17985 · HTML: https://mlsanity.com/paper/2509.17985/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 22 Sep 2025.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model](https://mlsanity.com/paper/2310.15110/) (2023)
- [Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures](https://mlsanity.com/paper/2211.07600/) (2022)
- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control](https://mlsanity.com/paper/2503.03751/) (2025)
- [TEXTure: Text-Guided Texturing of 3D Shapes](https://mlsanity.com/paper/2302.01721/) (2023)
- [LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias](https://mlsanity.com/paper/2410.17242/) (2024)
- [Stable Virtual Camera: Generative View Synthesis with Diffusion Models](https://mlsanity.com/paper/2503.14489/) (2025)
- [ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model](https://mlsanity.com/paper/2408.16767/) (2024)
- [Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models](https://mlsanity.com/paper/2312.13913/) (2023)
- [Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise](https://mlsanity.com/paper/2501.08331/) (2025)
- [MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views](https://mlsanity.com/paper/2411.04924/) (2024)
- [You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scale](https://mlsanity.com/paper/2412.06699/) (2024)
- [ControlRoom3D: Room Generation using Semantic Proxy Rooms](https://mlsanity.com/paper/2312.05208/) (2023)
- [Generic 3D Diffusion Adapter Using Controlled Multi-View Editing](https://mlsanity.com/paper/2403.12032/) (2024)
- [TEXGen: a Generative Diffusion Model for Mesh Textures](https://mlsanity.com/paper/2411.14740/) (2024)
- [FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesis](https://mlsanity.com/paper/2502.08244/) (2025)
- [TexPainter: Generative Mesh Texturing with Multi-view Consistency](https://mlsanity.com/paper/2406.18539/) (2024)
- [Coin3D: Controllable and Interactive 3D Assets Generation with Proxy-Guided Conditioning](https://mlsanity.com/paper/2405.08054/) (2024)
- [Phidias: A Generative Model for Creating 3D Content from Text, Image, and 3D Conditions with Reference-Augmented Diffusion](https://mlsanity.com/paper/2409.11406/) (2024)
- [Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior](https://mlsanity.com/paper/2404.06780/) (2024)

## Built on by (most cited first, 8 of 8)

- [Rethinking Video Generation Model for the Embodied World](https://mlsanity.com/paper/2601.15282/) (2026)
- [Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding](https://mlsanity.com/paper/2603.19235/) (2026)
- [VistaGEN: Consistent Driving Video Generation with Fine-Grained Control Using Multiview Visual-Language Reasoning](https://mlsanity.com/paper/2603.28353/) (2026)
- [Control-DINO: Feature Space Conditioning for Controllable Image-to-Video Diffusion](https://mlsanity.com/paper/2604.01761/) (2026)
- [From Seeing to Simulating: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluation](https://mlsanity.com/paper/2604.15805/) (2026)
- [YingVideo-MV: Music-Driven Multi-Stage Video Generation](https://mlsanity.com/paper/2512.02492/) (2025)
- [Explicit Layer Modeling for Video Object Insertion and Video Layer Decomposition](https://mlsanity.com/paper/2607.25802/) (2026)
- [World in World: Explore the World with World Models](https://mlsanity.com/paper/2609.11548/) (2026)

## Abstract (arXiv)

In this paper, we propose VideoFrom3D, a novel framework for synthesizing high-quality 3D scene videos from coarse geometry, a camera trajectory, and a reference image. Our approach streamlines the 3D graphic design workflow, enabling flexible design exploration and rapid production of deliverables. A straightforward approach to synthesizing a video from coarse geometry might condition a video diffusion model on geometric structure. However, existing video diffusion models struggle to generate high-fidelity results for complex scenes due to the difficulty of jointly modeling visual quality, motion, and temporal consistency. To address this, we propose a generative framework that leverages the complementary strengths of image and video diffusion models. Specifically, our framework consists of a Sparse Anchor-view Generation (SAG) and a Geometry-guided Generative Inbetweening (GGI) module. The SAG module generates high-quality, cross-view consistent anchor views using an image diffusion model, aided by Sparse Appearance-guided Sampling. Building on these anchor views, GGI module faithfully interpolates intermediate frames using a video diffusion model, enhanced by flow-based camera control and structural guidance. Notably, both modules operate without any paired dataset of 3D scene models and natural images, which is extremely difficult to obtain. Comprehensive experiments show that our method produces high-quality, style-consistent scene videos under diverse and challenging scenarios, outperforming simple and extended baselines.