# VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generation

arXiv: https://arxiv.org/abs/2502.07531 · HTML: https://mlsanity.com/paper/2502.07531/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 81 of its 122 references that are in the map.
- First posted on arXiv 11 Feb 2025.

## Builds on (most cited first, 30 of 81)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024)
- [Open-Sora: Democratizing Efficient Video Production for All](https://mlsanity.com/paper/2412.20404/) (2024)
- [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://mlsanity.com/paper/2312.03641/) (2023)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [VideoCrafter1: Open Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2310.19512/) (2023)
- [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023)
- [VideoComposer: Compositional Video Synthesis with Motion Controllability](https://mlsanity.com/paper/2306.02018/) (2023)
- [CameraCtrl: Enabling Camera Control for Text-to-Video Generation](https://mlsanity.com/paper/2404.02101/) (2024)
- [ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis](https://mlsanity.com/paper/2409.02048/) (2024)
- [DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory](https://mlsanity.com/paper/2308.08089/) (2023)
- [SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction](https://mlsanity.com/paper/2310.20700/) (2023)
- [Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling](https://mlsanity.com/paper/2401.15977/) (2024)
- [Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control](https://mlsanity.com/paper/2501.03847/) (2025)
- [Motion Prompting: Controlling Video Generation with Motion Trajectories](https://mlsanity.com/paper/2412.02700/) (2024)
- [SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models](https://mlsanity.com/paper/2311.16933/) (2023)
- [Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion](https://mlsanity.com/paper/2402.03162/) (2024)
- [CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation](https://mlsanity.com/paper/2406.02509/) (2024)
- [Tora: Trajectory-oriented Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2407.21705/) (2024)
- [DragAnything: Motion Control for Anything using Entity Representation](https://mlsanity.com/paper/2403.07420/) (2024)
- [VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control](https://mlsanity.com/paper/2407.12781/) (2024)

## Built on by (most cited first, 25 of 25)

- [ReCamMaster: Camera-Controlled Generative Rendering from A Single Video](https://mlsanity.com/paper/2503.11647/) (2025)
- [Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generation](https://mlsanity.com/paper/2504.14899/) (2025)
- [MotionStream: Real-Time Video Generation with Interactive Motion Controls](https://mlsanity.com/paper/2511.01266/) (2025)
- [Towards Understanding Camera Motions in Any Video](https://mlsanity.com/paper/2504.15376/) (2025)
- [ATI: Any Trajectory Instruction for Controllable Video Generation](https://mlsanity.com/paper/2505.22944/) (2025)
- [VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control](https://mlsanity.com/paper/2601.05138/) (2026)
- [EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidance](https://mlsanity.com/paper/2505.21876/) (2025)
- [AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories](https://mlsanity.com/paper/2602.14941/) (2026)
- [BulletTime: Decoupled Control of Time and Camera Pose for Video Generation](https://mlsanity.com/paper/2512.05076/) (2025)
- [Light-X: Generative 4D Video Rendering with Camera and Illumination Control](https://mlsanity.com/paper/2512.05115/) (2025)
- [World-R1: Reinforcing 3D Constraints for Text-to-Video Generation](https://mlsanity.com/paper/2604.24764/) (2026)
- [Frame In-N-Out: Unbounded Controllable Image-to-Video Generation](https://mlsanity.com/paper/2505.21491/) (2025)
- [SymphoMotion: Joint Control of Camera Motion and Object Dynamics for Coherent Video Generation](https://mlsanity.com/paper/2604.03723/) (2026)
- [IDCNet: Guided Video Diffusion for Metric-Consistent RGBD Scene Generation with Precise Camera Control](https://mlsanity.com/paper/2508.04147/) (2025)
- [LAMP: Language-Assisted Motion Planning for Controllable Video Generation](https://mlsanity.com/paper/2512.03619/) (2025)
- [OmniView: An All-Seeing Diffusion Model for 3D and 4D View Synthesis](https://mlsanity.com/paper/2512.10940/) (2025)
- [Stereo World Model: Camera-Guided Stereo Video Generation](https://mlsanity.com/paper/2603.17375/) (2026)
- [MotiMotion: Motion-Controlled Video Generation with Visual Reasoning](https://mlsanity.com/paper/2605.22818/) (2026)
- [Auteur: Language-Driven Cinematographic Framing for Human-Centric Video Generation](https://mlsanity.com/paper/2606.01900/) (2026)
- [Streaming Video Generation with Streaming Force Control](https://mlsanity.com/paper/2606.07508/) (2026)
- [Beyond Inpainting: Unleash 3D Understanding for Precise Camera-Controlled Video Generation](https://mlsanity.com/paper/2601.10214/) (2026)
- [UniGeo: Unifying Geometric Guidance for Camera-Controllable Image Editing via Video Models](https://mlsanity.com/paper/2604.17565/) (2026)
- [Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis](https://mlsanity.com/paper/2606.01590/) (2026)
- [UniMoCa: Unifying Motion and Camera Controls as Visual Proxies for Faithful Human Video Generation](https://mlsanity.com/paper/2608.01944/) (2026)
- [4Director: Controlling Video World Models with Rigid 3D Geometry](https://mlsanity.com/paper/2610.02160/) (2026)

## Abstract (arXiv)

Controllable image-to-video (I2V) generation transforms a reference image into a coherent video guided by user-specified control signals. While precise control over camera motion, object motion, and lighting is essential for high-fidelity creation, existing methods often treat these factors independently. This overlooks the physical coupling among viewpoint, geometry, and illumination in dynamic scenes, leading to visual inconsistencies such as mismatched shadows and perspective drift under simultaneous changes. We present VidCRAFT3, a unified and flexible I2V framework that explicitly models cross-factor interactions among geometry, motion, and illumination, enabling both independent and joint control over camera motion, object motion, and lighting direction. Image2Cloud provides explicit 3D geometric priors for accurate camera motion control. ObjMotionNet encodes sparse object trajectories into multi-scale motion features to guide realistic object motion. A Spatial Triple-Attention Transformer integrates lighting direction through lighting cross-attention for consistent relighting. To address the scarcity of jointly annotated data, we construct the VideoLightingDirection (VLD) dataset with accurate per-frame lighting direction annotations, and introduce a three-stage progressive training strategy that enables robust learning without fully joint annotations. Extensive experiments demonstrate that VidCRAFT3 achieves state-of-the-art performance in control precision and visual coherence across diverse scenarios.