mlsanityData to 7 Oct 2026

ShareGPT4Video: Improving Video Understanding and Generation with Better Captions

arXiv 2406.04325 ↗First posted 6 Jun 2024cs.CVcited by 386

Not on the map. Why?

Cited by 159 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present the ShareGPT4Video series, aiming to facilitate the video understanding of large video-language models (LVLMs) and the video generation of text-to-video models (T2VMs) via dense and precise captions. The series comprises: 1) ShareGPT4Video, 40K GPT4V annotated dense captions of videos with various lengths and sources, developed through carefully designed data filtering and annotating strategy. 2) ShareCaptioner-Video, an efficient and capable captioning model for arbitrary videos, with 4.8M high-quality aesthetic videos annotated by it. 3) ShareGPT4Video-8B, a simple yet superb LVLM that reached SOTA performance on three advancing video benchmarks. To achieve this, taking aside the non-scalable costly human annotators, we find using GPT4V to caption video with a naive multi-frame or frame-concatenation input strategy leads to less detailed and sometimes temporal-confused results. We argue the challenge of designing a high-quality video captioning strategy lies in three aspects: 1) Inter-frame precise temporal change understanding. 2) Intra-frame detailed content description. 3) Frame-number scalability for arbitrary-length videos. To this end, we meticulously designed a differential video captioning strategy, which is stable, scalable, and efficient for generating captions for videos with arbitrary resolution, aspect ratios, and length. Based on it, we construct ShareGPT4Video, which contains 40K high-quality videos spanning a wide range of categories, and the resulting captions encompass rich world knowledge, object attributes, camera movements, and crucially, detailed and precise temporal descriptions of events. Based on ShareGPT4Video, we further develop ShareCaptioner-Video, a superior captioner capable of efficiently generating high-quality captions for arbitrary videos...

Key facts. Cited by 386 papers in the map (data through 7 Oct 2026). Builds on 64 of its 90 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

64 of 90 references.

References · 64

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
38 more
  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  3. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  5. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  6. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  7. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  8. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  12. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  13. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  14. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  15. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  16. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  17. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  18. 2024Yi: Open Foundation Models by 01.AIcited by 706
  19. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  20. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  21. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  22. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  23. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  24. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  25. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  26. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  27. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  28. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  29. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  30. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  31. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  32. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  33. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  34. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  35. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  36. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  37. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  38. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 77; 2025: 207; 2026: 102.

386 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  4. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  5. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  6. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  7. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  8. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  9. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  10. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  11. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  12. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
38 more
  1. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  2. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  3. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  4. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  5. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  6. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  7. 2026Advancing Open-source World Modelscited by 144
  8. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  9. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  10. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 135
  11. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 134
  12. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  13. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
  14. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  15. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  16. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 99
  17. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 96
  18. 2025Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generationcited by 95
  19. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
  20. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 87
  21. 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuningcited by 86
  22. 2025Scaling RL to Long Videoscited by 83
  23. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 82
  24. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  25. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  26. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 78
  27. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 74
  28. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  29. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understandingcited by 72
  30. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 65
  31. 2024VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selectioncited by 64
  32. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  33. 2025SIM-CoT: Supervised Implicit Chain-of-Thoughtcited by 63
  34. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
  35. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 62
  36. 2025TinyLLaVA-Video-R1: Towards Smaller LMMs for Video Reasoningcited by 59
  37. 2024Online Video Understanding: OVBench and VideoChat-Onlinecited by 58
  38. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 58

50 most cited of 386 All 386 (JSON)