mlsanityData to 7 Oct 2026

ConCLVD: Controllable Chinese Landscape Video Generation via Diffusion Model

arXiv 2404.12903 ↗First posted 19 Apr 2024cs.MMcited by 0

On the map: Diffusion models → Video generation

Abstract (arXiv)

Chinese landscape painting is a gem of Chinese cultural and artistic heritage that showcases the splendor of nature through the deep observations and imaginations of its painters. Limited by traditional techniques, these artworks were confined to static imagery in ancient times, leaving the dynamism of landscapes and the subtleties of artistic sentiment to the viewer's imagination. Recently, emerging text-to-video (T2V) diffusion methods have shown significant promise in video generation, providing hope for the creation of dynamic Chinese landscape paintings. However, challenges such as the lack of specific datasets, the intricacy of artistic styles, and the creation of extensive, high-quality videos pose difficulties for these models in generating Chinese landscape painting videos. In this paper, we propose CLV-HD (Chinese Landscape Video-High Definition), a novel T2V dataset for Chinese landscape painting videos, and ConCLVD (Controllable Chinese Landscape Video Diffusion), a T2V model that utilizes Stable Diffusion. Specifically, we present a motion module featuring a dual attention mechanism to capture the dynamic transformations of landscape imageries, alongside a noise adapter to leverage unsupervised contrastive learning in the latent space. Following the generation of keyframes, we employ optical flow for frame interpolation to enhance video smoothness. Our method not only retains the essence of the landscape painting imageries but also achieves dynamic transitions, significantly advancing the field of artistic video generation. The source code and dataset are available at https://anonymous.4open.science/r/ConCLVD-EFE3.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 28 of its 44 references that are in the map. First posted on arXiv 19 Apr 2024.

Builds on

28 of 44 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  2. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation17 Apr
  3. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  4. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  5. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov

Other references · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  8. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  9. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  12. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
11 more
  1. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 153
  2. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
  3. 2021Asymmetric Bilateral Motion Estimation for Video Frame Interpolationcited by 101
  4. 2018To Create What You Tell: Generating Videos from Captionscited by 84
  5. 2023AutoDiffusion: Training-Free Optimization of Time Steps and Architectures for Automated Diffusion Model Accelerationcited by 70
  6. 2020SeCo: Exploring Sequence Supervision for Unsupervised Representation Learningcited by 65
  7. 2016Sync-DRAW: Automatic Video Generation using Deep Recurrent Attentive Architecturescited by 44
  8. 2017Attentive Semantic Video Generation using Captionscited by 28
  9. 2020End-to-End Chinese Landscape Painting Creation Using Generative Adversarial Networkscited by 18
  10. 2023CCLAP: Controllable Chinese Landscape Painting Generation via Latent Diffusion Modelcited by 5
  11. 2023Learning to Generate Poetic Chinese Landscape Painting with Calligraphycited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A New Chinese Landscape Paintings Generation Model based on Stable Diffusion using DreamBooth
  2. 2025Sketch-based Fluid Video Generation Using Motion-Guided Diffusion Models in Still Landscape Images
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Models
  4. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  5. 2023Space Narrative: Generating Images and 3D Scenes of Chinese Garden from Text using Deep Learning
  6. 2026FrescoDiffusion: 4K Image-to-Video with Prior-Regularized Tiled Diffusion
  7. 2022Video Diffusion Models
  8. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applications