mlsanityData to 7 Oct 2026

Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control

arXiv 2501.03847 ↗First posted 7 Jan 2025cs.CV cs.AI cs.GRcited by 184

On the map: Diffusion models → Video generation

Abstract (arXiv)

Diffusion models have demonstrated impressive performance in generating high-quality videos from text prompts or images. However, precise control over the video generation process, such as camera manipulation or content editing, remains a significant challenge. Existing methods for controlled video generation are typically limited to a single control type, lacking the flexibility to handle diverse control demands. In this paper, we introduce Diffusion as Shader (DaS), a novel approach that supports multiple video control tasks within a unified architecture. Our key insight is that achieving versatile video control necessitates leveraging 3D control signals, as videos are fundamentally 2D renderings of dynamic 3D content. Unlike prior methods limited to 2D control signals, DaS leverages 3D tracking videos as control inputs, making the video diffusion process inherently 3D-aware. This innovation allows DaS to achieve a wide range of video controls by simply manipulating the 3D tracking videos. A further advantage of using 3D tracking videos is their ability to effectively link frames, significantly enhancing the temporal consistency of the generated videos. With just 3 days of fine-tuning on 8 H800 GPUs using less than 10k videos, DaS demonstrates strong control capabilities across diverse tasks, including mesh-to-video generation, camera control, motion transfer, and object manipulation.

Key facts. Cited by 184 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 52 of its 76 references that are in the map. First posted on arXiv 7 Jan 2025.

Builds on

52 of 76 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024Video Motion Transfer with Diffusion Transformers10 Dec
  2. 2024CPA: Camera-pose-awareness Diffusion Transformer for Video Generation2 Dec
  3. 2024Trajectory Attention for Fine-grained Video Motion Control28 Nov
  4. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers27 Nov
  5. 2024Compositional 3D-aware Video Generation with LLM Director31 Aug
  6. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  7. 2024MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model30 May
  8. 2024Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance21 Mar
  9. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion5 Feb
  10. 2023TrailBlazer: Trajectory Control for Diffusion-Based Video Generation31 Dec
  11. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models28 Nov
  12. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
5 more
  1. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors18 Oct
  2. 2023CCEdit: Creative and Controllable Video Editing via Diffusion Models28 Sep
  3. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesis27 Apr
  4. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  5. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generation23 Nov

Other references · 35

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2023Segment Anythingcited by 8,158
  4. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  5. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  6. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  7. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  8. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  9. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  10. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  11. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  12. 2023TEXTure: Text-Guided Texturing of 3D Shapescited by 303
23 more
  1. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  2. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 273
  3. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 260
  4. 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modelingcited by 209
  5. 2018Stereo Magnification: Learning View Synthesis using Multiplane Imagescited by 206
  6. 2024SpatialTracker: Tracking Any 2D Pixels in 3D Spacecited by 178
  7. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 176
  8. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155
  9. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
  10. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124
  11. 2024ID-Animator: Zero-Shot Identity-Preserving Human Video Generationcited by 122
  12. 2024Boximator: Generating Rich and Controllable Motions for Video Synthesiscited by 107
  13. 2024Align3R: Aligned Monocular Depth Estimation for Dynamic Videoscited by 79
  14. 2023PEEKABOO: Interactive Video Generation via Masked-Diffusioncited by 76
  15. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  16. 2024Image Conductor: Precision Control for Interactive Video Synthesiscited by 62
  17. 2023Diffusion Handles: Enabling 3D Edits for Diffusion Models by Lifting Activations to 3Dcited by 59
  18. 2024GeoDiffuser: Geometry-Based Image Editing with Diffusion Modelscited by 34
  19. 2024AnimateAnything: Consistent and Controllable Animation for Video Generationcited by 34
  20. 2024I2VControl: Disentangled and Unified Video Motion Synthesis Controlcited by 19
  21. 2023Fine-grained Controllable Video Generation via Object Appearance and Contextcited by 18
  22. 2023Breathing New Life into 3D Assets with Generative Repaintingcited by 17
  23. 2025ProTracker: Probabilistic Integration for Robust and Accurate Point Trackingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 79; 2026: 105.

184 papers.

Sort:

By area:

  1. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
  2. 2025Video World Models with Long-term Spatial Memorycited by 124
  3. 2025Controllable Video Generation: A Surveycited by 93
  4. 2025SpatialTrackerV2: 3D Point Tracking Made Easycited by 84
  5. 2025Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generationcited by 75
  6. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
  7. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 60
  8. 2025Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidancecited by 45
  9. 2025CamCloneMaster: Enabling Reference-based Camera Control for Video Generationcited by 44
  10. 2025PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generationcited by 39
  11. 2026NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videoscited by 39
  12. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 37
38 more
  1. 2025GSFixer: Improving 3D Gaussian Splatting with Reference-Guided Video Diffusion Priorscited by 36
  2. 2025Reangle-A-Video: 4D Video Generation as Video-to-Video Translationcited by 35
  3. 2025Spatia: Video Generation with Updatable Spatial Memorycited by 33
  4. 2025EX-4D: EXtreme Viewpoint 4D Video Synthesis via Depth Watertight Meshcited by 32
  5. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 26
  6. 2025VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generationcited by 25
  7. 2025LongVie: Multimodal-Guided Controllable Ultra-Long Video Generationcited by 21
  8. 2025Generative Video Motion Editing with 3D Point Trackscited by 21
  9. 2025RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learningcited by 20
  10. 2025EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidancecited by 20
  11. 2026BridgeV2W: Bridging Video Generation Models to Embodied World Models via Embodiment Maskscited by 19
  12. 2026INSPATIO-WORLD: A Real-Time 4D World Simulator via Spatiotemporal Autoregressive Modelingcited by 18
  13. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  14. 2026WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memoriescited by 15
  15. 2025FlashDepth: Real-time Streaming Video Depth Estimation at 2K Resolutioncited by 14
  16. 2025CMD: Controllable Multiview Diffusion for 3D Editing and Progressive Generationcited by 14
  17. 2025Taming Video Models for 3D and 4D Generation via Zero-Shot Camera Controlcited by 14
  18. 2025Position: Interactive Generative Video as Next-Generation Game Enginecited by 13
  19. 20253D Scene Prompting for Scene-Consistent Camera-Controllable Video Generationcited by 13
  20. 2025Video-As-Prompt: Unified Semantic Control for Video Generationcited by 13
  21. 2025MotionV2V: Editing Motion in a Videocited by 13
  22. 2026AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memoriescited by 13
  23. 2025Dynamic View Synthesis as an Inverse Problemcited by 12
  24. 2025ManipDreamer3D : Synthesizing Plausible Robotic Manipulation Video with Occupancy-aware 3D Trajectorycited by 12
  25. 2025VFXMaster: Unlocking Dynamic Visual Effect Generation via In-Context Learningcited by 12
  26. 2025MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agentcited by 11
  27. 2025See4D: Pose-Free 4D Generation via Auto-Regressive Video Inpaintingcited by 11
  28. 2025DualCamCtrl: Dual-Branch Diffusion Model for Geometry-Aware Camera-Controlled Video Generationcited by 11
  29. 2025Light-X: Generative 4D Video Rendering with Camera and Illumination Controlcited by 11
  30. 2025Frame In-N-Out: Unbounded Controllable Image-to-Video Generationcited by 10
  31. 2025O-DisCo-Edit: Object Distortion Control for Unified Realistic Video Editingcited by 10
  32. 20254DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Modelscited by 10
  33. 2026LaVR: Scene Latent Conditioned Generative Video Trajectory Re-Rendering using Large 4D Reconstruction Modelscited by 10
  34. 2026Track4World: Feedforward World-centric Dense 3D Tracking of All Pixelscited by 10
  35. 2025I2V3D: Controllable image-to-video generation with 3D guidancecited by 9
  36. 2025PhysChoreo: Physics-Controllable Video Generation with Part-Aware Semantic Groundingcited by 9
  37. 2025LongVie 2: Multimodal Controllable Ultra-Long Video World Modelcited by 9
  38. 2025How Much 3D Do Video Foundation Models Encode?cited by 9

50 most cited of 184 All 184 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models
  2. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control
  3. 2026Lighting-grounded Video Generation with Renderer-based Agent Reasoning
  4. 2026ViDS: Video Diffusion Shader using 3D Face Tracking
  5. 2025X2Video: Adapting Diffusion Models for Multimodal Controllable Neural Video Rendering
  6. 2025CtrlVDiff: Controllable Video Generation via Unified Multimodal Video Diffusion
  7. 2024World-consistent Video Diffusion with Explicit 3D Modeling
  8. 2024Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model