mlsanityData to 7 Oct 2026

Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models

arXiv 2312.01409 ↗First posted 3 Dec 2023cs.CV cs.AI cs.GRcited by 29

On the map: Diffusion models → Video generation

Abstract (arXiv)

Traditional 3D content creation tools empower users to bring their imagination to life by giving them direct control over a scene's geometry, appearance, motion, and camera path. Creating computer-generated videos, however, is a tedious manual process, which can be automated by emerging text-to-video diffusion models. Despite great promise, video diffusion models are difficult to control, hindering a user to apply their own creativity rather than amplifying it. To address this challenge, we present a novel approach that combines the controllability of dynamic 3D meshes with the expressivity and editability of emerging diffusion models. For this purpose, our approach takes an animated, low-fidelity rendered mesh as input and injects the ground truth correspondence information obtained from the dynamic mesh into various stages of a pre-trained text-to-image generation model to output high-quality and temporally consistent frames. We demonstrate our approach on various examples where motion can be obtained by animating rigged assets or changing the camera path.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 38 of its 42 references that are in the map. First posted on arXiv 3 Dec 2023.

Builds on

38 of 42 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation1 Sep
  2. 2023ModelScope Text-to-Video Technical Report12 Aug
  3. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  4. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation13 Jun
  5. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  6. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  7. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  8. 2023Pix2Video: Video Editing using Image Diffusion22 Mar
  9. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  10. 2023Video Probabilistic Diffusion Models in Projected Latent Space15 Feb
  11. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  12. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
3 more
  1. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  2. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  3. 2022Video Diffusion Models7 Apr

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Classifier-Free Diffusion Guidancecited by 5,717
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  5. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  6. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  7. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  8. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  9. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  10. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  11. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  12. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
11 more
  1. 2023TEXTure: Text-Guided Texturing of 3D Shapescited by 303
  2. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  3. 2023An Edit Friendly DDPM Noise Space: Inversion and Manipulationscited by 243
  4. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  5. 2023Cross-Image Attention for Zero-Shot Appearance Transfercited by 135
  6. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  7. 2023CoDeF: Content Deformation Fields for Temporally Consistent Video Processingcited by 108
  8. 2023Compositional 3D Scene Generation using Locally Conditioned Diffusioncited by 101
  9. 2023TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Modelscited by 86
  10. 2022DiffDreamer: Towards Consistent Unsupervised Single-view Scene Extrapolation with Conditional Diffusion Modelscited by 51
  11. 2023Breathing New Life into 3D Assets with Generative Repaintingcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 13; 2025: 10; 2026: 5.

29 papers.

Sort:

By area:

  1. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  2. 2024Human4DiT: 360-degree Human Video Generation with 4D Diffusion Transformercited by 61
  3. 2024MotionCraft: Physics-based Zero-Shot Video Generationcited by 46
  4. 2024Diffusion Self-Distillation for Zero-Shot Customized Image Generationcited by 46
  5. 2024EG4D: Explicit Generation of 4D Object without Score Distillationcited by 45
  6. 2024AnimateAnything: Consistent and Controllable Animation for Video Generationcited by 34
  7. 2025Dynamic Concepts Personalization from Single Videoscited by 20
  8. 2024InterDyn: Controllable Interactive Dynamics with Video Diffusion Modelscited by 11
  9. 2025DualCamCtrl: Dual-Branch Diffusion Model for Geometry-Aware Camera-Controlled Video Generationcited by 11
  10. 2024Flash-Split: 2D Reflection Removal with Flash Cues and Latent Diffusion Separationcited by 10
  11. 2025I2V3D: Controllable image-to-video generation with 3D guidancecited by 9
  12. 2025Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxycited by 8
17 more
  1. 2025Vid-CamEdit: Video Camera Trajectory Editing with Generative Rendering from Estimated Geometrycited by 7
  2. 2025GenHSI: Controllable Generation of Human-Scene Interaction Videoscited by 6
  3. 2024Infinite-Resolution Integral Noise Warping for Diffusion Modelscited by 5
  4. 2024DissolveStereo: Coarse Depth Injection for Zero-Shot Stereo Video Generationcited by 5
  5. 2025GPS as a Control Signal for Image Generationcited by 5
  6. 2025Video4DGen: Enhancing Video and 4D Generation through Mutual Optimizationcited by 5
  7. 2023LatentMan: Generating Consistent Animated Characters using Image Diffusion Modelscited by 4
  8. 2024Learning Coherent Matrixized Representation in Latent Space for Volumetric 4D Generationcited by 4
  9. 2026EditCtrl: Disentangled Local and Global Control for Real-Time Generative Video Editingcited by 4
  10. 2024Tex4D: Zero-shot 4D Scene Texturing with Video Diffusion Modelscited by 2
  11. 2026Video Models as Native 4D Renderers: World-Grounded Conditioning from Animated Meshcited by 2
  12. 2026TourPhysics: Bringing Physics to World Models for Exploration and Manipulation from a Single Imagecited by 2
  13. 2025Advances in 4D Generation: A Surveycited by 1
  14. 2025MotionFlow:Learning Implicit Motion Flow for Complex Camera Trajectory Control in Video Generationcited by 1
  15. 2024Motion Dreamer: Boundary Conditional Motion Reasoning for Physically Coherent Video Generationcited by 0
  16. 2026MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Controlcited by 0
  17. 2026AgentGarten: Code Worlds for Evolving Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control
  2. 2026Efficient Camera-Controlled Video Generation of Static Scenes via Sparse Diffusion and 3D Rendering
  3. 2026Lighting-grounded Video Generation with Renderer-based Agent Reasoning
  4. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control
  5. 2026ActionMesh: Animated 3D Mesh Generation with Temporal 3D Diffusion
  6. 2025VideoFrom3D: 3D Scene Video Generation via Complementary Image and Video Diffusion Models
  7. 2026Towards 3D-Aware Video Diffusion Models: Render-Free Human Motion Control with Mesh Tokenization
  8. 2025Drive Any Mesh: 4D Latent Diffusion for Mesh Deformation from Video