# Attentive Semantic Video Generation using Captions

arXiv: https://arxiv.org/abs/1708.05980 · HTML: https://mlsanity.com/paper/1708.05980/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Aug 2017.

## Built on by (most cited first, 28 of 28)

- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [Conditional Image-to-Video Generation with Latent Flow Diffusion Models](https://mlsanity.com/paper/2303.13744/) (2023)
- [Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising](https://mlsanity.com/paper/2305.18264/) (2023)
- [SimDA: Simple Diffusion Adapter for Efficient Video Generation](https://mlsanity.com/paper/2308.09710/) (2023)
- [Make It Move: Controllable Image-to-Video Generation with Text Descriptions](https://mlsanity.com/paper/2112.02815/) (2021)
- [A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?](https://mlsanity.com/paper/2303.11717/) (2023)
- [ARTboldsymbol·V: Auto-Regressive Text-to-Video Generation with Diffusion Models](https://mlsanity.com/paper/2311.18834/) (2023)
- [Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning](https://mlsanity.com/paper/2203.02573/) (2022)
- [Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generation](https://mlsanity.com/paper/2211.12824/) (2022)
- [3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation](https://mlsanity.com/paper/2403.09439/) (2024)
- [AI-Generated Content (AIGC) for Various Data Modalities: A Survey](https://mlsanity.com/paper/2308.14177/) (2023)
- [Learning Semantic-Aware Dynamics for Video Prediction](https://mlsanity.com/paper/2104.09762/) (2021)
- [VideoDreamer: Customized Multi-Subject Text-to-Video Generation with Disen-Mix Finetuning on Language-Video Foundation Models](https://mlsanity.com/paper/2311.00990/) (2023)
- [Action2video: Generating Videos of Human 3D Actions](https://mlsanity.com/paper/2111.06925/) (2021)
- [ManipDreamer: Boosting Robotic Manipulation World Model with Action Tree and Visual Guidance](https://mlsanity.com/paper/2504.16464/) (2025)
- [M3L: Language-based Video Editing via Multi-Modal Multi-Level Transformers](https://mlsanity.com/paper/2104.01122/) (2021)
- [Text-to-Image Cross-Modal Generation: A Systematic Review](https://mlsanity.com/paper/2401.11631/) (2024)
- [Text-driven Video Prediction](https://mlsanity.com/paper/2210.02872/) (2022)
- [Text-guided 3D Human Generation from 2D Collections](https://mlsanity.com/paper/2305.14312/) (2023)
- [Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing Reliability,Reproducibility, and Practicality](https://mlsanity.com/paper/2406.08845/) (2024)
- [Video Generation from Text Employing Latent Path Construction for Temporal Modeling](https://mlsanity.com/paper/2107.13766/) (2021)
- [Disentangling Content and Motion for Text-Based Neural Video Manipulation](https://mlsanity.com/paper/2211.02980/) (2022)
- [Xp-GAN: Unsupervised Multi-object Controllable Video Generation](https://mlsanity.com/paper/2111.10233/) (2021)
- [Autoencoding Video Latents for Adversarial Video Generation](https://mlsanity.com/paper/2201.06888/) (2022)
- [ConCLVD: Controllable Chinese Landscape Video Generation via Diffusion Model](https://mlsanity.com/paper/2404.12903/) (2024)

## Abstract (arXiv)

This paper proposes a network architecture to perform variable length semantic video generation using captions. We adopt a new perspective towards video generation where we allow the captions to be combined with the long-term and short-term dependencies between video frames and thus generate a video in an incremental manner. Our experiments demonstrate our network architecture's ability to distinguish between objects, actions and interactions in a video and combine them to generate videos for unseen captions. The network also exhibits the capability to perform spatio-temporal style transfer when asked to generate videos for a sequence of captions. We also show that the network's ability to learn a latent representation allows it generate videos in an unsupervised manner and perform other tasks such as action recognition. (Accepted in International Conference in Computer Vision (ICCV) 2017)