mlsanityData to 7 Oct 2026

AnimateAnything: Consistent and Controllable Animation for Video Generation

arXiv 2411.10836 ↗First posted 16 Nov 2024cs.CVcited by 34

Not on the map. Why?

Abstract (arXiv)

We present a unified controllable video generation approach AnimateAnything that facilitates precise and consistent video manipulation across various conditions, including camera trajectories, text prompts, and user motion annotations. Specifically, we carefully design a multi-scale control feature fusion network to construct a common motion representation for different conditions. It explicitly converts all control information into frame-by-frame optical flows. Then we incorporate the optical flows as motion priors to guide final video generation. In addition, to reduce the flickering issues caused by large-scale motion, we propose a frequency-based stabilization module. It can enhance temporal coherence by ensuring the video's frequency domain consistency. Experiments demonstrate that our method outperforms the state-of-the-art approaches. For more details and videos, please refer to the webpage: https://yu-shaonian.github.io/Animate_Anything/.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 49 of its 61 references that are in the map. First posted on arXiv 16 Nov 2024.

Builds on

49 of 61 references.

References · 49

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  8. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  9. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  10. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  11. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  12. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
37 more
  1. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  2. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  3. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  4. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  5. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  6. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  7. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  8. 2023CoTracker: It is Better to Track Togethercited by 516
  9. 2023DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Visioncited by 514
  10. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  11. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  12. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  13. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 354
  14. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
  15. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 330
  16. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 324
  17. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  18. 2022Unifying Flow, Stereo and Depth Estimationcited by 270
  19. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 260
  20. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Predictioncited by 231
  21. 2023Visual Geometry Grounded Deep Structure From Motioncited by 225
  22. 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modelingcited by 209
  23. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generationcited by 163
  24. 2024Semantic Image Inversion and Editing using Rectified Stochastic Differential Equationscited by 162
  25. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  26. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  27. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 104
  28. 2022ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wildcited by 90
  29. 2024MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Modelcited by 87
  30. 2023PEEKABOO: Interactive Video Generation via Masked-Diffusioncited by 76
  31. 2022DiffDreamer: Towards Consistent Unsupervised Single-view Scene Extrapolation with Conditional Diffusion Modelscited by 51
  32. 2024MagicTime: Time-lapse Video Generation Models as Metamorphic Simulatorscited by 51
  33. 2023Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Modelscited by 29
  34. 2020Sound2Sight: Generating Visual Dynamics from Sound and Contextcited by 28
  35. 2022FNeVR: Neural Volume Rendering for Face Animationcited by 22
  36. 2019Self-Supervised Learning via Conditional Motion Propagationcited by 20
  37. 2023FocusFlow: Boosting Key-Points Optical Flow Estimation for Autonomous Drivingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 21; 2026: 11.

34 papers.

Sort:

By area:

  1. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 184
  2. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  3. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
  4. 2025Towards Understanding Camera Motions in Any Videocited by 62
  5. 2025FullDiT: Multi-Task Video Generative Foundation Model with Full Attentioncited by 45
  6. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 37
  7. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  8. 2025Omni-Effects: Unified and Spatially-Controllable Visual Effects Generationcited by 27
  9. 2025Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representationcited by 22
  10. 2025Video-As-Prompt: Unified Semantic Control for Video Generationcited by 13
  11. 2025EchoVideo: Identity-Preserving Human Video Generation by Multimodal Feature Fusioncited by 12
  12. 2024Motion Modes: What Could Happen Next?cited by 11
22 more
  1. 2025MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agentcited by 11
  2. 2025AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Modelscited by 10
  3. 2025IDCNet: Guided Video Diffusion for Metric-Consistent RGBD Scene Generation with Precise Camera Controlcited by 7
  4. 2025MotionRAG: Motion Retrieval-Augmented Image-to-Video Generationcited by 7
  5. 2025The World is Your Canvas: Painting Promptable Events with Reference Images, Trajectories, and Textcited by 7
  6. 2026Future Optical Flow Prediction Improves Robot Control & Video Generationcited by 7
  7. 2025DropletVideo: A Dataset and Approach to Explore Integral Spatio-Temporal Consistent Video Generationcited by 6
  8. 2025Streaming Drag-Oriented Interactive Video Manipulation: Drag Anything, Anytime!cited by 5
  9. 2025Learning to Generate Rigid Body Interactions with Video Diffusion Modelscited by 2
  10. 2025UniMMVSR: A Unified Multi-Modal Framework for Cascaded Video Super-Resolutioncited by 1
  11. 2025Consistent text-to-image generation via scene de-contextualizationcited by 1
  12. 2025V-Warper: Appearance-Consistent Video Diffusion Personalization via Value Warpingcited by 1
  13. 2026Focal Guidance: Unlocking Controllability from Semantic-Weak Layers in Video Diffusion Modelscited by 1
  14. 2026FlowC2S: Flowing from Current to Succeeding Frames for Fast and Memory-Efficient Video Continuationcited by 0
  15. 2026MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Controlcited by 0
  16. 2026GraphVid: Interactive Graph-Controllable Video Generationcited by 0
  17. 2026FlexComposer: Unified Video Compositing from Images to Dynamic Footage with Flexible Trajectory Controlcited by 0
  18. 2026C3-UniMM: Causal Cycle-Consistent Unified Multimodal Modeling via Super Alignment and Shared Decoding Spacecited by 0
  19. 2026SNF-Bench: Separating Static Drift from Natural Flow in Long-Horizon Fixed-Camera Video Generationcited by 0
  20. 2026Streaming Video Editing with Easy Adaptationcited by 0
  21. 2026MotionSpec: Spectral Trajectory Supervision for Motion-Consistent Video Generationcited by 0
  22. 20264Director: Controlling Video World Models with Rigid 3D Geometrycited by 0