mlsanityData to 14 Oct 2026

Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos

arXiv 2304.01186 ↗First posted 3 Apr 2023cs.CVcited by 314

Not on the map. Why?

Cited by 116 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generating text-editable and pose-controllable character videos have an imperious demand in creating various digital human. Nevertheless, this task has been restricted by the absence of a comprehensive dataset featuring paired video-pose captions and the generative prior models for videos. In this work, we design a novel two-stage training scheme that can utilize easily obtained datasets (i.e.,image pose pair and pose-free video) and the pre-trained text-to-image (T2I) model to obtain the pose-controllable character videos. Specifically, in the first stage, only the keypoint-image pairs are used only for a controllable text-to-image generation. We learn a zero-initialized convolutional encoder to encode the pose information. In the second stage, we finetune the motion of the above network via a pose-free video dataset by adding the learnable temporal self-attention and reformed cross-frame self-attention blocks. Powered by our new designs, our method successfully generates continuously pose-controllable character videos while keeps the editing and concept composition ability of the pre-trained T2I model. The code and models will be made publicly available.

Key facts. Cited by 314 papers in the map (data through 14 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 3 Apr 2023.

Builds on

36 of 50 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2022Classifier-Free Diffusion Guidancecited by 5,847
  8. 2022Generative Adversarial Networkscited by 5,627
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  10. 2017Neural Discrete Representation Learningcited by 4,712
  11. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
24 more
  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  2. 2022Video Diffusion Modelscited by 2,305
  3. 2014Conditional Generative Adversarial Netscited by 2,162
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  5. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  7. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  8. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  9. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  10. 2020First Order Motion Model for Image Animationcited by 704
  11. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  12. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 466
  13. 2022Blended Latent Diffusioncited by 432
  14. 2018Video-to-Video Synthesiscited by 390
  15. 2018Everybody Dance Nowcited by 315
  16. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  17. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 256
  18. 2021Motion Representations for Articulated Animationcited by 255
  19. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  20. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  21. 2019Few-shot Video-to-Video Synthesiscited by 197
  22. 2019Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesiscited by 161
  23. 2022FineDance: A Fine-grained Choreography Dataset for 3D Full Body Dance Generationcited by 88
  24. 2022SimVTP: Simple Video Text Pre-training with Masked Autoencoderscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 37; 2024: 86; 2025: 115; 2026: 76.

314 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  2. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  3. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 688
  4. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
  5. 2025VACE: All-in-One Video Creation and Editingcited by 479
  6. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
  7. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  8. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Modelcited by 330
  9. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 327
  10. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 285
  11. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 247
  12. 2023A Survey on Video Diffusion Modelscited by 206
38 more
  1. 2024Taming Rectified Flow for Inversion and Editingcited by 204
  2. 2024MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidancecited by 198
  3. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 175
  4. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 174
  5. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 173
  6. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 167
  7. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  8. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 158
  9. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 154
  10. 2023DisCo: Disentangled Control for Realistic Human Dance Generationcited by 142
  11. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 139
  12. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
  13. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 120
  14. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 108
  15. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 102
  16. 2024DiT4Edit: Diffusion Transformer for Image Editingcited by 101
  17. 2025Controllable Video Generation: A Surveycited by 98
  18. 2023Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Groupingcited by 92
  19. 2023Animate-A-Story: Storytelling with Retrieval-Augmented Video Generationcited by 90
  20. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
  21. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 86
  22. 2025Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuningcited by 81
  23. 2025OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generationcited by 73
  24. 2023Reti-Diff: Illumination Degradation Image Restoration with Retinex-based Latent Diffusion Modelcited by 69
  25. 2024Inducing High Energy-Latency of Large Vision-Language Models with Verbose Imagescited by 69
  26. 2024ToonCrafter: Generative Cartoon Interpolationcited by 68
  27. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 67
  28. 2023InstructEdit: Improving Automatic Masks for Diffusion-based Image Editing With User Instructionscited by 64
  29. 2024MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modelingcited by 64
  30. 2025Any2AnyTryon: Leveraging Adaptive Position Embeddings for Versatile Virtual Clothing Taskscited by 63
  31. 2023MagicStick: Controllable Video Editing via Control Handle Transformationscited by 62
  32. 2025Follow-Your-Creation: Empowering 4D Creation through Video Inpaintingcited by 62
  33. 2024Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitivescited by 61
  34. 2023Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Modelscited by 59
  35. 2024COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editingcited by 59
  36. 2025HunyuanPortrait: Implicit Condition Control for Enhanced Portrait Animationcited by 59
  37. 2023MotionEditor: Editing Video Motion via Content-Aware Diffusioncited by 57
  38. 2025RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Controlcited by 57

50 most cited of 314 All 314 (JSON)