mlsanityData to 7 Oct 2026

Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models

arXiv 2306.05424 ↗First posted 8 Jun 2023cs.CVcited by 1,047

Not on the map. Why?

Cited by 450 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Conversation agents fueled by Large Language Models (LLMs) are providing a new way to interact with visual data. While there have been initial attempts for image-based conversation models, this work addresses the under-explored field of video-based conversation by introducing Video-ChatGPT. It is a multimodal model that merges a video-adapted visual encoder with an LLM. The resulting model is capable of understanding and generating detailed conversations about videos. We introduce a new dataset of 100,000 video-instruction pairs used to train Video-ChatGPT acquired via manual and semi-automated pipeline that is easily scalable and robust to label noise. We also develop a quantitative evaluation framework for video-based dialogue models to objectively analyze the strengths and weaknesses of video-based dialogue models. Code: https://github.com/mbzuai-oryx/Video-ChatGPT.

Key facts. Cited by 1,047 papers in the map (data through 7 Oct 2026). Builds on 29 of its 40 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

29 of 40 references.

References · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  12. 2023VideoChat: Chat-Centric Video Understandingcited by 851
17 more
  1. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  2. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  3. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  4. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  5. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  6. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  7. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  8. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  9. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  10. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  11. 2022Fine-tuned CLIP Models are Efficient Video Learnerscited by 178
  12. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 145
  13. 2021Probabilistic two-stage detectioncited by 130
  14. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
  15. 2021Class-agnostic Object Detection with Multi-modal Transformercited by 85
  16. 2023Tag2Text: Guiding Vision-Language Model via Image Taggingcited by 84
  17. 2021Towards General Purpose Vision Systemscited by 46

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 74; 2024: 318; 2025: 430; 2026: 225.

1,047 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  5. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  7. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  8. 2023A Survey on Multimodal Large Language Modelscited by 898
  9. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  10. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  11. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  12. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
38 more
  1. 2023A Comprehensive Overview of Large Language Modelscited by 639
  2. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  3. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  4. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  5. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  6. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  7. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  8. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  9. 2024Long Context Transfer from Language to Visioncited by 502
  10. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  11. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  12. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  13. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  14. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  15. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  16. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  17. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  18. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  19. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  20. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  21. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  22. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensingcited by 320
  23. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  24. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  25. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  26. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  27. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  28. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  29. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  30. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  31. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  32. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  33. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  34. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  35. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 248
  36. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  37. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  38. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226

50 most cited of 1,047 All 1,047 (JSON)