mlsanityData to 7 Oct 2026

Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise

arXiv 2501.08331 ↗First posted 14 Jan 2025cs.CVcited by 114

On the map: Diffusion models → Video generation

Abstract (arXiv)

Generative modeling aims to transform random noise into structured outputs. In this work, we enhance video diffusion models by allowing motion control via structured latent noise sampling. This is achieved by just a change in data: we pre-process training videos to yield structured noise. Consequently, our method is agnostic to diffusion model design, requiring no changes to model architectures or training pipelines. Specifically, we propose a novel noise warping algorithm, fast enough to run in real time, that replaces random temporal Gaussianity with correlated warped noise derived from optical flow fields, while preserving the spatial Gaussianity. The efficiency of our algorithm enables us to fine-tune modern video diffusion base models using warped noise with minimal overhead, and provide a one-stop solution for a wide range of user-friendly motion control: local object motion control, global camera movement control, and motion transfer. The harmonization between temporal coherence and spatial Gaussianity in our warped noise leads to effective motion control while maintaining per-frame pixel quality. Extensive experiments and user studies demonstrate the advantages of our method, making it a robust and scalable approach for controlling motion in video diffusion models. Video results are available on our webpage: https://eyeline-labs.github.io/Go-with-the-Flow. Source code and model checkpoints are available on GitHub: https://github.com/Eyeline-Labs/Go-with-the-Flow.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 67 of its 76 references that are in the map. First posted on arXiv 14 Jan 2025.

Builds on

67 of 76 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  2. 2024FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models24 Jun
  3. 2024Training-free Camera Control for Video Generation14 Jun
  4. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation4 Jun
  5. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control27 May
  6. 2024ReVideo: Remake a Video with Motion and Content Control22 May
  7. 2024MotionCraft: Physics-based Zero-Shot Video Generation22 May
  8. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation2 May
  9. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generation2 Apr
  10. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion5 Feb
  11. 2024Latte: Latent Diffusion Transformer for Video Generation5 Jan
  12. 2023Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation7 Dec
8 more
  1. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models28 Nov
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  3. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generation30 Oct
  4. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors18 Oct
  5. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation9 Oct
  6. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  7. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  8. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr

Other references · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  9. 2022Scalable Diffusion Models with Transformerscited by 5,931
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
35 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  4. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  5. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  6. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  7. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  8. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  9. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  10. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  11. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  12. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  13. 2023CoTracker: It is Better to Track Togethercited by 516
  14. 2023DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Visioncited by 514
  15. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  16. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  17. 2024MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motioncited by 431
  18. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 395
  19. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 392
  20. 2023ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shiftingcited by 360
  21. 2018Learning Blind Video Temporal Consistencycited by 247
  22. 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modelingcited by 209
  23. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 176
  24. 2024CAT4D: Create Anything in 4D with Multi-View Video Diffusion Modelscited by 160
  25. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  26. 2023WonderJourney: Going from Anywhere to Everywherecited by 157
  27. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124
  28. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 104
  29. 2023PEEKABOO: Interactive Video Generation via Masked-Diffusioncited by 76
  30. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  31. 2024ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuningcited by 68
  32. 2024Image Conductor: Precision Control for Interactive Video Synthesiscited by 62
  33. 2024DifFRelight: Diffusion-Based Facial Performance Relightingcited by 35
  34. 2024Infinite-Resolution Integral Noise Warping for Diffusion Modelscited by 5
  35. 2023Scalable Motion Style Transfer with Constrained Diffusion Generationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 46; 2026: 68.

114 papers.

Sort:

By area:

  1. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  2. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
  3. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 60
  4. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 49
  5. 2025Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidancecited by 45
  6. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 43
  7. 2025NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamicscited by 40
  8. 2025PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generationcited by 39
  9. 2025Reangle-A-Video: 4D Video Generation as Video-to-Video Translationcited by 35
  10. 2026Inference-time Physics Alignment of Video Generative Models with Latent World Modelscited by 32
  11. 2025LoRA-Edit: Controllable First-Frame-Guided Video Editing via Mask-Aware LoRA Fine-Tuningcited by 27
  12. 2025ContextFlow: Training-Free Video Object Editing via Adaptive Context Enrichmentcited by 27
38 more
  1. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 26
  2. 2025LongVie: Multimodal-Guided Controllable Ultra-Long Video Generationcited by 21
  3. 2025Generative Video Motion Editing with 3D Point Trackscited by 21
  4. 2025ShapeGen4D: Towards High Quality 4D Shape Generation from Videoscited by 20
  5. 2026RealWonder: Real-Time Physical Action-Conditioned Video Generationcited by 18
  6. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 17
  7. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  8. 2026WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memoriescited by 15
  9. 2026Vista4D: Video Reshooting with 4D Point Cloudscited by 15
  10. 2025RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformerscited by 14
  11. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoisingcited by 13
  12. 2025MotionV2V: Editing Motion in a Videocited by 13
  13. 2026AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memoriescited by 13
  14. 2025BulletTime: Decoupled Control of Time and Camera Pose for Video Generationcited by 12
  15. 2026Physical Simulator In-the-Loop Video Generationcited by 12
  16. 2026LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Modelcited by 12
  17. 2025Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Modelscited by 11
  18. 2025Can Video Diffusion Model Reconstruct 4D Geometry?cited by 10
  19. 2026Track4World: Feedforward World-centric Dense 3D Tracking of All Pixelscited by 10
  20. 2025I2V3D: Controllable image-to-video generation with 3D guidancecited by 9
  21. 2025First Frame Is the Place to Go for Video Content Customizationcited by 9
  22. 2025LongVie 2: Multimodal Controllable Ultra-Long Video World Modelcited by 9
  23. 2026PerpetualWonder: Long-Horizon Action-Conditioned 4D Scene Generationcited by 9
  24. 2026VOID: Video Object and Interaction Deletioncited by 9
  25. 2026PhysInOne: Visual Physics Learning and Reasoning in One Suitecited by 9
  26. 2025VideoFrom3D: 3D Scene Video Generation via Complementary Image and Video Diffusion Modelscited by 8
  27. 2025Target-Aware Video Diffusion Modelscited by 7
  28. 2025MoGAN: Improving Motion Quality in Video Diffusion via Few-Step Motion Adversarial Post-Trainingcited by 7
  29. 2026Future Optical Flow Prediction Improves Robot Control & Video Generationcited by 7
  30. 2026UCM: Unified Modeling of Camera Control and Memory with Time-aware Positional Encoding Warping for World Modelscited by 7
  31. 2026Masked Visual Actions for Unified World Modelingcited by 7
  32. 2025Product of Experts for Visual Generationcited by 6
  33. 2025GenHSI: Controllable Generation of Human-Scene Interaction Videoscited by 6
  34. 2025FlexTraj: Image-to-Video Generation with Flexible Point Trajectory Controlcited by 6
  35. 2025TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixelscited by 6
  36. 2026VII: Visual Instruction Injection for Jailbreaking Image-to-Video Generation Modelscited by 6
  37. 2026Prompt Relay: Inference-Time Temporal Control for Multi-Event Video Generationcited by 6
  38. 2026World-R1: Reinforcing 3D Constraints for Text-to-Video Generationcited by 6

50 most cited of 114 All 114 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Controllable Longer Image Animation with Diffusion Models
  2. 2024Noise Crystallization and Liquid Noise: Zero-shot Video Generation using Image Diffusion Models
  3. 2025FlowLoss: Dynamic Flow-Conditioned Loss Strategy for Video Diffusion Models
  4. 2025Taming Video Models for 3D and 4D Generation via Zero-Shot Camera Control
  5. 2025Tuning-Free Long Video Generation via Global-Local Collaborative Diffusion
  6. 2026Efficient Video Diffusion Models: Advancements and Challenges
  7. 2024Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based Video Generation
  8. 2023Control-A-Video: Controllable Text-to-Video Diffusion Models with Motion Prior and Reward Feedback Learning