mlsanityData to 7 Oct 2026

Intelligent Director: An Automatic Framework for Dynamic Visual Composition using ChatGPT

arXiv 2402.15746 ↗First posted 24 Feb 2024cs.CV cs.AI cs.MMcited by 1

Not on the map. Why?

Abstract (arXiv)

With the rise of short video platforms represented by TikTok, the trend of users expressing their creativity through photos and videos has increased dramatically. However, ordinary users lack the professional skills to produce high-quality videos using professional creation software. To meet the demand for intelligent and user-friendly video creation tools, we propose the Dynamic Visual Composition (DVC) task, an interesting and challenging task that aims to automatically integrate various media elements based on user requirements and create storytelling videos. We propose an Intelligent Director framework, utilizing LENS to generate descriptions for images and video frames and combining ChatGPT to generate coherent captions while recommending appropriate music names. Then, the best-matched music is obtained through music retrieval. Then, materials such as captions, images, videos, and music are integrated to seamlessly synthesize the video. Finally, we apply AnimeGANv2 for style transfer. We construct UCF101-DVC and Personal Album datasets and verified the effectiveness of our framework in solving DVC through qualitative and quantitative comparisons, along with user studies, demonstrating its substantial potential.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 33 references that are in the map. First posted on arXiv 24 Feb 2024.

Builds on

23 of 33 references.

References · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  8. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2022Video Diffusion Modelscited by 2,280
  11. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  12. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
11 more
  1. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  2. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  3. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  4. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  5. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  6. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  7. 2023Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animatorcited by 74
  8. 2023Towards Language Models That Can See: Computer Vision Through the LENS of Natural Languagecited by 53
  9. 2023DirecT2V: Large Language Models are Frame-Level Directors for Zero-Shot Text-to-Video Generationcited by 47
  10. 2023MovieFactory: Automatic Movie Creation from Text using Large Generative Models for Language and Imagescited by 42
  11. 2020Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrievalcited by 27

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Compositional 3D-aware Video Generation with LLM Directorcited by 18