mlsanityData to 14 Oct 2026

Multimodal Cinematic Video Synthesis Using Text-to-Image and Audio Generation Models

arXiv 2506.10005 ↗First posted 6 Apr 2025cs.CV cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Advances in generative artificial intelligence have altered multimedia creation, allowing for automatic cinematic video synthesis from text inputs. This work describes a method for creating 60-second cinematic movies incorporating Stable Diffusion for high-fidelity image synthesis, GPT-2 for narrative structuring, and a hybrid audio pipeline using gTTS and YouTube-sourced music. It uses a five-scene framework, which is augmented by linear frame interpolation, cinematic post-processing (e.g., sharpening), and audio-video synchronization to provide professional-quality results. It was created in a GPU-accelerated Google Colab environment using Python 3.11. It has a dual-mode Gradio interface (Simple and Advanced), which supports resolutions of up to 1024x768 and frame rates of 15-30 FPS. Optimizations such as CUDA memory management and error handling ensure reliability. The experiments demonstrate outstanding visual quality, narrative coherence, and efficiency, furthering text-to-video synthesis for creative, educational, and industrial applications.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 12 of its 15 references that are in the map. First posted on arXiv 6 Apr 2025.

Builds on

12 of 15 references.

References · 12

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  3. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  4. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  5. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  6. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 455
  7. 2023A Survey on Video Diffusion Modelscited by 206
  8. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 41
  9. 2023Advancements in Generative AI: A Comprehensive Review of GANs, GPT, Autoencoders, Diffusion Model, and Transformerscited by 36
  10. 2024Grid Diffusion Models for Text-to-Video Generationcited by 21
  11. 2024SSE: Multimodal Semantic Data Selection and Enrichment for Industrial-scale Data Assimilationcited by 7
  12. 2024BGDB: Bernoulli-Gaussian Decision Block with Improved Denoising Diffusion Probabilistic Modelscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Captain Cinema: Towards Short Movie Generation
  2. 2025Fine-Tuning Open Video Generators for Cinematic Scene Synthesis: A Small-Data Pipeline with LoRA and Wan2.1 I2V
  3. 2023MovieFactory: Automatic Movie Creation from Text using Large Generative Models for Language and Images
  4. 2023AADiff: Audio-Aligned Video Synthesis with Text-to-Image Diffusion
  5. 2025Seedance 1.5 pro: A Native Audio-Visual Joint Generation Foundation Model
  6. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation
  7. 2026Diffusion Models for Joint Audio-Video Generation
  8. 2024Can video generation replace cinematographers? Research on the cinematic language of generated video