mlsanityData to 7 Oct 2026

ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding System

arXiv 2304.14407 ↗First posted 27 Apr 2023cs.CVcited by 65

Not on the map. Why?

Abstract (arXiv)

Existing deep video models are limited by specific tasks, fixed input-output spaces, and poor generalization capabilities, making it difficult to deploy them in real-world scenarios. In this paper, we present our vision for multimodal and versatile video understanding and propose a prototype system, system. Our system is built upon a tracklet-centric paradigm, which treats tracklets as the basic video unit and employs various Video Foundation Models (ViFMs) to annotate their properties e.g., appearance, motion, etc. All the detected tracklets are stored in a database and interact with the user through a database manager. We have conducted extensive case studies on different types of in-the-wild videos, which demonstrates the effectiveness of our method in answering various video-related problems. Our project is available at https://www.wangjunke.info/ChatVideo/

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Builds on 39 of its 46 references that are in the map. First posted on arXiv 27 Apr 2023.

Builds on

39 of 46 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
27 more
  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  2. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  3. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  4. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  5. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  6. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  7. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  8. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  9. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  10. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  11. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  12. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  13. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  14. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  15. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  16. 2019TVQA+: Spatio-Temporal Grounding for Video Question Answeringcited by 153
  17. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  18. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  19. 2022Towards Grand Unification of Object Trackingcited by 93
  20. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learningcited by 90
  21. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 76
  22. 2021Efficient Video Transformers with Spatial-Temporal Token Selectioncited by 58
  23. 2021UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Trainingcited by 43
  24. 2023Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptionscited by 42
  25. 2022Look Before You Match: Instance Understanding Matters in Video Object Segmentationcited by 33
  26. 2023OmniTracker: Unifying Object Tracking by Tracking-with-Detectioncited by 19
  27. 2023Streaming Video Modelcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 30; 2025: 15; 2026: 6.

65 papers.

Sort:

By area:

  1. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  2. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  3. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  4. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  5. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  6. 2023Video Understanding with Large Language Models: A Surveycited by 203
  7. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  8. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  9. 2024MMedAgent: Learning to Use Medical Tools with Multi-modal Agentcited by 108
  10. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  11. 2023ADriver-I: A General World Model for Autonomous Drivingcited by 102
  12. 2024Large Multimodal Agents: A Surveycited by 99
38 more
  1. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 89
  2. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  3. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answeringcited by 73
  4. 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactionscited by 67
  5. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 64
  6. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  7. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  8. 2023Graph Prompt Learning: A Comprehensive Survey and Beyondcited by 61
  9. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 56
  10. 2024Foundation Models for Video Understanding: A Surveycited by 46
  11. 2024V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuningcited by 45
  12. 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 44
  13. 2024Training-free Video Temporal Grounding using Large-scale Pre-trained Modelscited by 28
  14. 2025CoS: Chain-of-Shot Prompting for Long Video Understandingcited by 27
  15. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 26
  16. 2024Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Modelscited by 24
  17. 2024OmniVid: A Generative Framework for Universal Video Understandingcited by 22
  18. 2024ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Modelscited by 21
  19. 2024Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understandingcited by 18
  20. 2024What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interactioncited by 18
  21. 2024ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videoscited by 18
  22. 2023ChatFace: Chat-Guided Real Face Editing via Diffusion Latent Space Manipulationcited by 16
  23. 2024VLLMs Provide Better Context for Emotion Understanding Through Common Sense Reasoningcited by 15
  24. 2026From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agentscited by 14
  25. 2025PVChat: Personalized Video Chat with One-Shot Learningcited by 13
  26. 2025Video-CoT: A Comprehensive Dataset for Spatiotemporal Understanding of Videos Based on Chain-of-Thoughtcited by 13
  27. 2024TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignmentcited by 10
  28. 2024LLaSA: A Sensor-Aware LLM for Natural Language Reasoning of Human Activity from IMU Datacited by 10
  29. 2023Long Video Understanding with Learnable Retrieval in Video-Language Modelscited by 9
  30. 2024Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language Modelscited by 9
  31. 2024Question-Instructed Visual Descriptions for Zero-Shot Video Question Answeringcited by 8
  32. 2024GDeR: Safeguarding Efficiency, Balancing, and Robustness via Prototypical Graph Pruningcited by 8
  33. 2026Semantic-Aware Adaptive Visual Memory for Streaming Video Understandingcited by 8
  34. 2024LLAVIDAL: A Large LAnguage VIsion Model for Daily Activities of Livingcited by 7
  35. 2025AViLA: Asynchronous Vision-Language Agent for Streaming Multimodal Data Interactioncited by 6
  36. 2025From Long Videos to Engaging Clips: A Human-Inspired Video Editing Framework with Multimodal Narrative Understandingcited by 6
  37. 2024FocusChat: Text-guided Long Video Understanding via Spatiotemporal Information Filteringcited by 5
  38. 2025SurgLLM: A Versatile Large Multimodal Model with Spatial Focus and Temporal Awareness for Surgical Video Understandingcited by 5

50 most cited of 65 All 65 (JSON)