# A Survey on Generative AI and LLM for Video Generation, Understanding, and Streaming

arXiv: https://arxiv.org/abs/2404.16038 · HTML: https://mlsanity.com/paper/2404.16038/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 84 of its 114 references that are in the map.
- First posted on arXiv 30 Jan 2024.

## Builds on (most cited first, 30 of 84)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [VideoPoet: A Large Language Model for Zero-Shot Video Generation](https://mlsanity.com/paper/2312.14125/) (2023)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)

## Built on by (most cited first, 30 of 37)

- [Mobile Edge Intelligence for Large Language Models: A Contemporary Survey](https://mlsanity.com/paper/2407.18921/) (2024)
- [Controllable Video Generation: A Survey](https://mlsanity.com/paper/2507.16869/) (2025)
- [A Comprehensive Survey of Foundation Models in Medicine](https://mlsanity.com/paper/2406.10729/) (2024)
- [AI-Generated Video Detection via Perceptual Straightening](https://mlsanity.com/paper/2507.00583/) (2025)
- [MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detection](https://mlsanity.com/paper/2503.18132/) (2025)
- [ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Models](https://mlsanity.com/paper/2507.09876/) (2025)
- [Generative AI for Cel-Animation: A Survey](https://mlsanity.com/paper/2501.06250/) (2025)
- [Achieving Time Series Reasoning Requires Rethinking Model Design, Tasks Formulation, and Evaluation](https://mlsanity.com/paper/2502.01477/) (2025)
- [The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective](https://mlsanity.com/paper/2407.08583/) (2024)
- [Do Language Models Understand Time?](https://mlsanity.com/paper/2412.13845/) (2024)
- [Beyond Training: Dynamic Token Merging for Zero-Shot Video Understanding](https://mlsanity.com/paper/2411.14401/) (2024)
- [DreamCinema: Cinematic Transfer with Free Camera and 3D Character](https://mlsanity.com/paper/2408.12601/) (2024)
- [QuickVideo: Real-Time Long Video Understanding with System Algorithm Co-Design](https://mlsanity.com/paper/2505.16175/) (2025)
- [SCBench: A Sports Commentary Benchmark for Video LLMs](https://mlsanity.com/paper/2412.17637/) (2024)
- [Video Is Worth a Thousand Images: Exploring the Latest Trends in Long Video Generation](https://mlsanity.com/paper/2412.18688/) (2024)
- [Scaling Laws in Scientific Discovery with AI and Robot Scientists](https://mlsanity.com/paper/2503.22444/) (2025)
- [DeepFleet: Multi-Agent Foundation Models for Mobile Robots](https://mlsanity.com/paper/2508.08574/) (2025)
- [A Survey: Spatiotemporal Consistency in Video Generation](https://mlsanity.com/paper/2502.17863/) (2025)
- [Neuroplasticity and Corruption in Model Mechanisms: A Case Study Of Indirect Object Identification](https://mlsanity.com/paper/2503.01896/) (2025)
- [Toxicity in Online Platforms and AI Systems: A Survey of Needs, Challenges, Mitigations, and Future Directions](https://mlsanity.com/paper/2509.25539/) (2025)
- [Watermarking Large Language Models and the Generated Content: Opportunities and Challenges](https://mlsanity.com/paper/2410.19096/) (2024)
- [ILDiff: Generate Transparent Animated Stickers by Implicit Layout Distillation](https://mlsanity.com/paper/2412.20901/) (2024)
- [LLMPopcorn: Exploring LLMs as Assistants for Popular Micro-video Generation](https://mlsanity.com/paper/2502.12945/) (2025)
- [On-device System of Compositional Multi-tasking in Large Language Models](https://mlsanity.com/paper/2510.13848/) (2025)
- [VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inference](https://mlsanity.com/paper/2603.29494/) (2026)
- [Generative AI for Video Trailer Synthesis: From Extractive Heuristics to Autoregressive Creativity](https://mlsanity.com/paper/2604.04953/) (2026)
- [CAPRAG: A Large Language Model Solution for Customer Service and Automatic Reporting using Vector and Graph Retrieval-Augmented Generation](https://mlsanity.com/paper/2501.13993/) (2025)
- [Cognitive Science-Inspired Evaluation of Core Capabilities for Object Understanding in AI](https://mlsanity.com/paper/2503.21668/) (2025)
- [Leveraging Large Language Models in Visual Speech Recognition: Model Scaling, Context-Aware Decoding, and Iterative Polishing](https://mlsanity.com/paper/2506.02012/) (2025)
- [SSG-Dit: A Spatial Signal Guided Framework for Controllable Video Generation](https://mlsanity.com/paper/2508.17062/) (2025)

## Abstract (arXiv)

This paper offers an insightful examination of how currently top-trending AI technologies, i.e., generative artificial intelligence (Generative AI) and large language models (LLMs), are reshaping the field of video technology, including video generation, understanding, and streaming. It highlights the innovative use of these technologies in producing highly realistic videos, a significant leap in bridging the gap between real-world dynamics and digital creation. The study also delves into the advanced capabilities of LLMs in video understanding, demonstrating their effectiveness in extracting meaningful information from visual content, thereby enhancing our interaction with videos. In the realm of video streaming, the paper discusses how LLMs contribute to more efficient and user-centric streaming experiences, adapting content delivery to individual viewer preferences. This comprehensive review navigates through the current achievements, ongoing challenges, and future possibilities of applying Generative AI and LLMs to video-related tasks, underscoring the immense potential these technologies hold for advancing the field of video technology related to multimedia, networking, and AI communities.