mlsanityData to 14 Oct 2026

HARIVO: Harnessing Text-to-Image Models for Video Generation

arXiv 2410.07763 ↗First posted 10 Oct 2024cs.CV cs.AIcited by 8

On the map: Diffusion models → Video generation

Abstract (arXiv)

We present a method to create diffusion-based video models from pretrained Text-to-Image (T2I) models. Recently, AnimateDiff proposed freezing the T2I model while only training temporal layers. We advance this method by proposing a unique architecture, incorporating a mapping network and frame-wise tokens, tailored for video generation while maintaining the diversity and creativity of the original T2I model. Key innovations include novel loss functions for temporal smoothness and a mitigating gradient sampling technique, ensuring realistic and temporally consistent video generation despite limited public video data. We have successfully integrated video-specific inductive biases into the architecture and loss functions. Our method, built on the frozen StableDiffusion model, simplifies training processes and allows for seamless integration with off-the-shelf models like ControlNet and DreamBooth. project page: https://kwonminki.github.io/HARIVO

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 35 of its 45 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

35 of 45 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023ModelScope Text-to-Video Technical Report12 Aug
  2. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  3. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  5. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  6. 2023Dreamix: Video Diffusion Models are General Video Editors2 Feb
  7. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generation23 Nov
  8. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  9. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  10. 2022Video Diffusion Models7 Apr

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  7. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  8. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  9. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  10. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  11. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  12. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
13 more
  1. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  2. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  3. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  4. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 727
  5. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 635
  6. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  7. 2022Diffusion Models already have a Semantic Latent Spacecited by 328
  8. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  9. 2023Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometrycited by 125
  10. 2022Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidancecited by 67
  11. 20223DDesigner: Towards Photorealistic 3D Object Generation and Editing with Text-guided Diffusion Modelscited by 57
  12. 2022Understanding DDPM Latent Codes Through Optimal Transportcited by 45
  13. 2023Highly Personalized Text Embedding for Image Manipulation by Stable Diffusioncited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 1.

8 papers.

  1. 2024Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generationcited by 44
  2. 2025Pursuing Temporal-Consistent Video Virtual Try-On via Dynamic Pose Interactioncited by 11
  3. 2025FinePhys: Fine-grained Human Action Generation by Explicitly Incorporating Physical Laws for Effective Skeletal Guidancecited by 9
  4. 2025Control and Realism: Best of Both Worlds in Layout-to-Image without Trainingcited by 3
  5. 2025MultiCOIN: Multi-Modal COntrollable Video INbetweeningcited by 2
  6. 2024PoM: Efficient Image and Video Generation with the Polynomial Mixercited by 1
  7. 2025FlowBlending: Stage-Aware Multi-Model Sampling for Fast and High-Fidelity Video Generationcited by 0
  8. 2026From Static to Dynamic: On-Policy Distillation from Image to Video Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  2. 2026Image-to-Video Diffusion: From Foundations to Open Frontiers
  3. 2024Grid Diffusion Models for Text-to-Video Generation
  4. 2023DreamVideo: High-Fidelity Image-to-Video Generation with Image Retention and Text Guidance
  5. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
  6. 2025Anchored Video Generation: Decoupling Scene Construction and Temporal Synthesis in Text-to-Video Diffusion Models
  7. 2025Video Text Preservation with Synthetic Text-Rich Videos
  8. 2025ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models