# Coarse-to-Real: Generative Rendering for Populated Dynamic Scenes arXiv: https://arxiv.org/abs/2601.22301 ยท HTML: https://mlsanity.com/paper/2601.22301/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 44 references that are in the map. - First posted on arXiv 29 Jan 2026. ## Builds on (most cited first, 28 of 28) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023) - [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023) - [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025) - [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022) - [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024) - [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022) - [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023) - [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023) - [VideoCrafter1: Open Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2310.19512/) (2023) - [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025) - [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://mlsanity.com/paper/2404.01291/) (2024) - [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018) - [GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control](https://mlsanity.com/paper/2503.03751/) (2025) - [MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation](https://mlsanity.com/paper/2212.09478/) (2022) - [World Simulation with Video Foundation Models for Physical AI](https://mlsanity.com/paper/2511.00062/) (2025) - [Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control](https://mlsanity.com/paper/2501.03847/) (2025) - [SpatialTracker: Tracking Any 2D Pixels in 3D Space](https://mlsanity.com/paper/2404.04319/) (2024) - [Motion Prompting: Controlling Video Generation with Motion Trajectories](https://mlsanity.com/paper/2412.02700/) (2024) - [Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models](https://mlsanity.com/paper/2303.17599/) (2023) - [VidToMe: Video Token Merging for Zero-Shot Video Editing](https://mlsanity.com/paper/2312.10656/) (2023) - [Motion-Conditioned Diffusion Model for Controllable Video Synthesis](https://mlsanity.com/paper/2304.14404/) (2023) - [Consistent Video-to-Video Transfer Using Synthetic Dataset](https://mlsanity.com/paper/2311.00213/) (2023) - [Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding](https://mlsanity.com/paper/2501.07888/) (2025) - [Control-A-Video: Controllable Text-to-Video Diffusion Models with Motion Prior and Reward Feedback Learning](https://mlsanity.com/paper/2305.13840/) (2023) - [Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance](https://mlsanity.com/paper/2512.08765/) (2025) - [Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models](https://mlsanity.com/paper/2312.12487/) (2023) - [WorldGen: From Text to Traversable and Interactive 3D Worlds](https://mlsanity.com/paper/2511.16825/) (2025) ## Built on by (most cited first, 9 of 9) - [Programmable World Model](https://mlsanity.com/paper/2609.10540/) (2026) - [Magpie: Real-Time World Renderer for Interactive Games](https://mlsanity.com/paper/2608.27168/) (2026) - [WorldRover: A Scalable Synthetic Video Data Engine for World Exploration with Rich Annotations](https://mlsanity.com/paper/2608.15659/) (2026) - [Closing the Loop: Training-Free Revisit Consistency for Autoregressive Generative Rendering](https://mlsanity.com/paper/2607.21848/) (2026) - [Diffuse the object, keep its label: curating detector training data from a few unlabeled photographs via VLM-built 3D vegetation scenes](https://mlsanity.com/paper/2608.09691/) (2026) - [Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models](https://mlsanity.com/paper/2610.01614/) (2026) - [Proxy2World: Learning to Generate Worlds From Lightweight Proxies without Seeing Them](https://mlsanity.com/paper/2609.35023/) (2026) - [ROWBench: Do Video Models Render What the Program Specifies?](https://mlsanity.com/paper/2610.02205/) (2026) - [AgentGarten: Code Worlds for Evolving Agents](https://mlsanity.com/paper/2610.12374/) (2026) ## Abstract (arXiv) Traditional rendering pipelines rely on complex assets, accurate materials and lighting, and substantial computational resources to produce realistic imagery, yet they still face challenges in scalability and realism for populated dynamic scenes. We present C2R (Coarse-to-Real), a generative rendering framework that synthesizes real-style urban crowd videos from coarse 3D simulations. Our approach uses coarse 3D renderings to explicitly control scene layout, camera motion, and human trajectories, while a learned neural renderer generates realistic appearance, lighting, and fine-scale dynamics guided by text prompts. To overcome the lack of paired training data between coarse simulations and real videos, we adopt a two-stage synthetic-real domain-hedging strategy that first learns a strong generative prior from large-scale real footage, and then introduces controllability by using a small amount of paired synthetic coarse-to-fine data to anchor shared implicit spatio-temporal features across domains. The resulting system supports coarse-to-fine control, generalizes across diverse CG and game inputs, and produces temporally consistent, controllable, and realistic urban scene videos from minimal 3D input. We will release the model and project webpage at https://gonzalognogales.github.io/coarse2real/.