mlsanityData to 7 Oct 2026

VideoChat: Chat-Centric Video Understanding

arXiv 2305.06355 ↗First posted 10 May 2023cs.CV cs.CLcited by 851

Not on the map. Why?

Cited by 350 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we initiate an attempt of developing an end-to-end chat-centric video understanding system, coined as VideoChat. It integrates video foundation models and large language models via a learnable neural interface, excelling in spatiotemporal reasoning, event localization, and causal relationship inference. To instructively tune this system, we build a video-centric instruction dataset, composed of thousands of videos associated with detailed descriptions and conversations. This dataset emphasizes spatiotemporal reasoning and captures causal relationships, providing a valuable asset for training our chat-centric video understanding system. Preliminary qualitative experiments demonstrate the potential of our system across a broad spectrum of video applications, which could serve as a simple prototype system for future research on chat-centric video understanding. Access our code and data at https://github.com/OpenGVLab/Ask-Anything

Key facts. Cited by 851 papers in the map (data through 7 Oct 2026). Builds on 52 of its 81 references that are in the map. First posted on arXiv 10 May 2023.

Builds on

52 of 81 references.

References · 52

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  3. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  4. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  5. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  6. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  7. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  8. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  9. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  10. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  11. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  12. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  13. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  14. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  15. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  16. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  17. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  18. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  19. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  20. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  21. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  22. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  23. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  24. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  25. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  26. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  27. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  28. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  29. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  30. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  31. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  32. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  33. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  34. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
  35. 2022UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormercited by 116
  36. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96
  37. 2023Tag2Text: Guiding Vision-Language Model via Image Taggingcited by 84
  38. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 104; 2024: 290; 2025: 310; 2026: 147.

851 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  5. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  6. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  7. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  9. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  10. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  11. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  12. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
38 more
  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  2. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  3. 2023A Survey on Multimodal Large Language Modelscited by 898
  4. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  5. 2023A Comprehensive Overview of Large Language Modelscited by 639
  6. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  7. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  8. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  9. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  10. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  11. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  12. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  13. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  14. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  15. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  16. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  17. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  18. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  19. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  20. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  21. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
  22. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  23. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  24. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  25. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  26. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  27. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  28. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  29. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  30. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  31. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  32. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  33. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  34. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  35. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  36. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  37. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  38. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 248

50 most cited of 851 All 851 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models
  2. 2025Know-Show: Benchmarking Video-Language Models on Spatio-Temporal Grounded Reasoning
  3. 2026VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding
  4. 2026Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Models
  5. 2025VCRBench: Exploring Long-form Causal Reasoning Capabilities of Large Video Language Models
  6. 2026Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events
  7. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoning
  8. 2026VideoSeeker: Incentivizing Instance-level Video Understanding via Native Agentic Tool Invocation