mlsanityData to 14 Oct 2026

SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation

arXiv 2411.04989 ↗First posted 7 Nov 2024cs.CV cs.LGcited by 68

Not on the map. Why?

Cited by 27 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Methods for image-to-video generation have achieved impressive, photo-realistic quality. However, adjusting specific elements in generated videos, such as object motion or camera movement, is often a tedious process of trial and error, e.g., involving re-generating videos with different random seeds. Recent techniques address this issue by fine-tuning a pre-trained model to follow conditioning signals, such as bounding boxes or point trajectories. Yet, this fine-tuning procedure can be computationally expensive, and it requires datasets with annotated object motion, which can be difficult to procure. In this work, we introduce SG-I2V, a framework for controllable image-to-video generation that is self-guidedunicodex2013offering zero-shot control by relying solely on the knowledge present in a pre-trained image-to-video diffusion model without the need for fine-tuning or external knowledge. Our zero-shot method outperforms unsupervised baselines while significantly narrowing down the performance gap with supervised models in terms of visual quality and motion fidelity. Additional details and video results are available on our project page: https://kmcode1.github.io/Projects/SG-I2V

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Builds on 77 of its 88 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

77 of 88 references.

References · 77

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  10. 2013Auto-Encoding Variational Bayescited by 4,053
  11. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  12. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
38 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  2. 2022Video Diffusion Modelscited by 2,305
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  4. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  5. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  6. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  7. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  8. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  9. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  10. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  11. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  12. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  13. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  14. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 727
  15. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 702
  16. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 688
  17. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  18. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  19. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  20. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
  21. 2023CoTracker: It is Better to Track Togethercited by 522
  22. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
  23. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  24. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  25. 2023Emergent Correspondence from Image Diffusioncited by 446
  26. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 388
  27. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Modelcited by 330
  28. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 313
  29. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  30. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  31. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 301
  32. 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editingcited by 294
  33. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 264
  34. 2023Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifoldcited by 219
  35. 2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
  36. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondencecited by 214
  37. 2023DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Modelscited by 195
  38. 2024InstanceDiffusion: Instance-level Control for Image Generationcited by 175

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 31; 2026: 31.

68 papers.

Sort:

By area:

  1. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 185
  2. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 178
  3. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 154
  4. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 117
  5. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 61
  6. 2025RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Controlcited by 57
  7. 2025Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signalscited by 53
  8. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 52
  9. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 45
  10. 2025Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidancecited by 45
  11. 2024LeviTor: 3D Trajectory Oriented Image-to-Video Synthesiscited by 39
  12. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
38 more
  1. 2025VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generationcited by 25
  2. 2026BridgeV2W: Bridging Video Generation Models to Embodied World Models via Embodiment Maskscited by 20
  3. 2024ObjCtrl-2.5D: Training-free Object Control with Camera Posescited by 18
  4. 2025GenDoP: Auto-regressive Camera Trajectory Generation as a Director of Photographycited by 18
  5. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  6. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applicationscited by 14
  7. 2025BulletTime: Decoupled Control of Time and Camera Pose for Video Generationcited by 14
  8. 2026Physical Simulator In-the-Loop Video Generationcited by 14
  9. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoisingcited by 13
  10. 2025Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Modelscited by 11
  11. 2025AnyI2V: Animating Any Conditional Image with Motion Controlcited by 11
  12. 2026PISCO: Precise Video Instance Insertion with Sparse Controlcited by 10
  13. 2024Learning Camera Movement Control from Real-World Drone Videoscited by 9
  14. 2025Zo3T: Zero-Shot 3D-Aware Trajectory-Guided Image-to-Video Generation via Test-Time Trainingcited by 9
  15. 2026Goal Force: Teaching Video Models To Accomplish Physics-Conditioned Goalscited by 9
  16. 2025LiON-LoRA: Rethinking LoRA Fusion to Unify Controllable Spatial and Temporal Generation for Video Diffusioncited by 7
  17. 2025The World is Your Canvas: Painting Promptable Events with Reference Images, Trajectories, and Textcited by 7
  18. 2026CtrlAttack: A Unified Attack on World-Model Control in Diffusion Modelscited by 7
  19. 2026HVG-3D: Bridging Real and Simulation Domains for 3D-Conditional Hand-Object Interaction Video Synthesiscited by 7
  20. 2025PanFlow: Decoupled Motion Control for Panoramic Video Generationcited by 6
  21. 2025On Equivariance and Fast Sampling in Video Diffusion Models Trained with Warped Noisecited by 5
  22. 2025Streaming Drag-Oriented Interactive Video Manipulation: Drag Anything, Anytime!cited by 5
  23. 2025STANCE: Motion Coherent Video Generation Via Sparse-to-Dense Anchored Encodingcited by 5
  24. 2025PoseAnything: Universal Pose-guided Video Generation with Part-aware Temporal Coherencecited by 5
  25. 2026FlashMotion: Few-Step Controllable Video Generation with Trajectory Guidancecited by 5
  26. 2026PhyCo: Learning Controllable Physical Priors for Generative Motioncited by 5
  27. 2025Physics-Grounded Motion Forecasting via Equation Discovery for Trajectory-Guided Image-to-Video Generationcited by 4
  28. 2026Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Jointscited by 4
  29. 2026DreamVideo-Omni: Omni-Motion Controlled Multi-Subject Video Customization with Latent Identity Reinforcement Learningcited by 4
  30. 2025TGT: Text-Grounded Trajectories for Locally Controlled Video Generationcited by 3
  31. 2026Motion Forcing: A Decoupled Framework for Robust Video Generation in Motion Dynamicscited by 3
  32. 2026Image-to-Video Diffusion: From Foundations to Open Frontierscited by 3
  33. 2026CP4D: Compositional Physics-aware 4D Scene Generationcited by 2
  34. 2026PhysStream: Streaming Physics-Grounded Video Generation with Structured Scene Memory and Fine-Grained Motion Controlcited by 2
  35. 2025SSG-Dit: A Spatial Signal Guided Framework for Controllable Video Generationcited by 1
  36. 2025Video Consistency Distance: Enhancing Temporal Consistency for Image-to-Video Generation via Reward-Based Fine-Tuningcited by 1
  37. 2025Generative Augmented Reality: Paradigms, Technologies, and Future Applicationscited by 1
  38. 2026TrajLoom: Dense Future Trajectory Generation from Videocited by 1

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling
  2. 2026PartiCam: Camera Controlled Video Generation with Reward Guidance
  3. 2024Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based Video Generation
  4. 2024Ctrl-V: Higher Fidelity Video Generation with Bounding-Box Controlled Object Motion
  5. 2025LMP: Leveraging Motion Prior in Zero-Shot Video Generation with Diffusion Transformer
  6. 2024Boosting Camera Motion Control for Video Diffusion Transformers
  7. 2025Training-Free Motion-Guided Video Generation with Enhanced Temporal Consistency Using Motion Consistency Loss
  8. 2025Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Think