mlsanityData to 7 Oct 2026

A Survey on Generative AI and LLM for Video Generation, Understanding, and Streaming

arXiv 2404.16038 ↗First posted 30 Jan 2024cs.CV cs.AI cs.MMcited by 37

Not on the map. Why?

Abstract (arXiv)

This paper offers an insightful examination of how currently top-trending AI technologies, i.e., generative artificial intelligence (Generative AI) and large language models (LLMs), are reshaping the field of video technology, including video generation, understanding, and streaming. It highlights the innovative use of these technologies in producing highly realistic videos, a significant leap in bridging the gap between real-world dynamics and digital creation. The study also delves into the advanced capabilities of LLMs in video understanding, demonstrating their effectiveness in extracting meaningful information from visual content, thereby enhancing our interaction with videos. In the realm of video streaming, the paper discusses how LLMs contribute to more efficient and user-centric streaming experiences, adapting content delivery to individual viewer preferences. This comprehensive review navigates through the current achievements, ongoing challenges, and future possibilities of applying Generative AI and LLMs to video-related tasks, underscoring the immense potential these technologies hold for advancing the field of video technology related to multimedia, networking, and AI communities.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 84 of its 114 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

84 of 114 references.

References · 84

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2023Segment Anythingcited by 8,158
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2022Video Diffusion Modelscited by 2,280
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  9. 2017The Kinetics Human Action Video Datasetcited by 1,894
  10. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  12. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
38 more
  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  3. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  4. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  5. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  6. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  7. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  8. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  9. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  10. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  11. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  12. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  13. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  14. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  15. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  16. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  17. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  18. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  19. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  20. 2016Generating Videos with Scene Dynamicscited by 376
  21. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  22. 2018Everybody Dance Nowcited by 313
  23. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 312
  24. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  25. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  26. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  27. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  28. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  29. 2018Stochastic Video Generation with a Learned Priorcited by 244
  30. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  31. 2022Learning Video Representations from Large Language Modelscited by 204
  32. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  33. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  34. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  35. 2017Decomposing Motion and Content for Natural Video Sequence Predictioncited by 168
  36. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 162
  37. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 161
  38. 2023DisCo: Disentangled Control for Realistic Human Dance Generationcited by 141

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 20; 2026: 6.

37 papers.

Sort:

By area:

  1. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  2. 2025Controllable Video Generation: A Surveycited by 93
  3. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  4. 2025AI-Generated Video Detection via Perceptual Straighteningcited by 34
  5. 2025MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detectioncited by 24
  6. 2025ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Modelscited by 18
  7. 2025Generative AI for Cel-Animation: A Surveycited by 17
  8. 2025Achieving Time Series Reasoning Requires Rethinking Model Design, Tasks Formulation, and Evaluationcited by 15
  9. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  10. 2024Do Language Models Understand Time?cited by 14
  11. 2024Beyond Training: Dynamic Token Merging for Zero-Shot Video Understandingcited by 13
  12. 2024DreamCinema: Cinematic Transfer with Free Camera and 3D Charactercited by 9
25 more
  1. 2024SCBench: A Sports Commentary Benchmark for Video LLMscited by 7
  2. 2024Video Is Worth a Thousand Images: Exploring the Latest Trends in Long Video Generationcited by 7
  3. 2025QuickVideo: Real-Time Long Video Understanding with System Algorithm Co-Designcited by 7
  4. 2025Scaling Laws in Scientific Discovery with AI and Robot Scientistscited by 5
  5. 2025A Survey: Spatiotemporal Consistency in Video Generationcited by 3
  6. 2025Neuroplasticity and Corruption in Model Mechanisms: A Case Study Of Indirect Object Identificationcited by 3
  7. 2025DeepFleet: Multi-Agent Foundation Models for Mobile Robotscited by 3
  8. 2025Toxicity in Online Platforms and AI Systems: A Survey of Needs, Challenges, Mitigations, and Future Directionscited by 3
  9. 2024Watermarking Large Language Models and the Generated Content: Opportunities and Challengescited by 2
  10. 2024ILDiff: Generate Transparent Animated Stickers by Implicit Layout Distillationcited by 2
  11. 2025LLMPopcorn: Exploring LLMs as Assistants for Popular Micro-video Generationcited by 2
  12. 2025On-device System of Compositional Multi-tasking in Large Language Modelscited by 2
  13. 2026VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inferencecited by 2
  14. 2026Generative AI for Video Trailer Synthesis: From Extractive Heuristics to Autoregressive Creativitycited by 2
  15. 2025CAPRAG: A Large Language Model Solution for Customer Service and Automatic Reporting using Vector and Graph Retrieval-Augmented Generationcited by 1
  16. 2025Cognitive Science-Inspired Evaluation of Core Capabilities for Object Understanding in AIcited by 1
  17. 2025Leveraging Large Language Models in Visual Speech Recognition: Model Scaling, Context-Aware Decoding, and Iterative Polishingcited by 1
  18. 2026DSIPA: Detecting LLM-Generated Texts via Sentiment-Invariant Patterns Divergence Analysiscited by 1
  19. 2024Decoupled Video Generation with Chain of Training-free Diffusion Model Expertscited by 0
  20. 2025Think-Before-Draw: Decomposing Emotion Semantics & Fine-Grained Controllable Expressive Talking Head Generationcited by 0
  21. 2025SSG-Dit: A Spatial Signal Guided Framework for Controllable Video Generationcited by 0
  22. 2025LLMs for LLMs: A Structured Prompting Methodology for Long Legal Documentscited by 0
  23. 2026DataCube: A Video Retrieval Platform via Natural Language Semantic Profilingcited by 0
  24. 2026Generative AI and Digital Ecosystem Resilience: A Proactive Lifecycle-Based Surveycited by 0
  25. 2026AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agentscited by 0