mlsanityData to 7 Oct 2026

TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding

arXiv 2312.02051 ↗First posted 4 Dec 2023cs.CV cs.AI cs.CLcited by 399

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

This work proposes TimeChat, a time-sensitive multimodal large language model specifically designed for long video understanding. Our model incorporates two key architectural contributions: (1) a timestamp-aware frame encoder that binds visual content with the timestamp of each frame, and (2) a sliding video Q-Former that produces a video token sequence of varying lengths to accommodate videos of various durations. Additionally, we construct an instruction-tuning dataset, encompassing 6 tasks and a total of 125K instances, to further enhance TimeChat's instruction-following performance. Experiment results across various video understanding tasks, such as dense captioning, temporal grounding, and highlight detection, demonstrate TimeChat's strong zero-shot temporal localization and reasoning capabilities. For example, it achieves +9.2 F1 score and +2.8 CIDEr on YouCook2, +5.8 HIT@1 on QVHighlights, and +27.5 R@1 (IoU=0.5) on Charades-STA, compared to state-of-the-art video large language models, holding the potential to serve as a versatile video assistant for long-form video comprehension tasks and satisfy realistic user requirements.

Key facts. Cited by 399 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 65 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

54 of 65 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun
  2. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning27 Feb

Other references · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
38 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  4. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  5. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  6. 2017Dense-Captioning Events in Videoscited by 789
  7. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  8. 2017Localizing Moments in Video with Natural Languagecited by 660
  9. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  10. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  11. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  12. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  13. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  14. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  15. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  16. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  17. 2023UniVTG: Towards Unified Video-Language Temporal Groundingcited by 189
  18. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  19. 2023Query-Dependent Video Representation for Moment Retrieval and Highlight Detectioncited by 166
  20. 2022UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detectioncited by 148
  21. 2016Video Summarization with Long Short-term Memorycited by 142
  22. 2021End-to-End Dense Video Captioning with Parallel Decodingcited by 133
  23. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  24. 2023FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generationcited by 119
  25. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  26. 2021Negative Sample Matters: A Renaissance of Metric Learning for Temporal Groundingcited by 90
  27. 2021Video Summarization Using Deep Neural Networks: A Surveycited by 85
  28. 2023Hierarchical Video-Moment Retrieval and Step-Captioningcited by 83
  29. 2020Multimodal Pretraining for Dense Video Captioningcited by 79
  30. 2021QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queriescited by 72
  31. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  32. 2023TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understandingcited by 49
  33. 2023StepFormer: Self-supervised Step Discovery and Localization in Instructional Videoscited by 37
  34. 2023Towards Generalisable Video Moment Retrieval: Visual-Dynamic Injection to Image-Text Pre-Trainingcited by 32
  35. 2022End-to-end Dense Video Captioning as Sequence Generationcited by 27
  36. 2023Movie101: A New Movie Understanding Benchmarkcited by 25
  37. 2021Learning Relation Alignment for Calibrated Cross-modal Retrievalcited by 21
  38. 2022Unifying Event Detection and Captioning as Sequence Generation via Pre-Trainingcited by 17

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 96; 2025: 181; 2026: 118.

399 papers.

Sort:

By area:

  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  2. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  3. 2024Long Context Transfer from Language to Visioncited by 502
  4. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  5. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  6. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  7. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  8. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  9. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  10. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  11. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  12. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
38 more
  1. 2023Video Understanding with Large Language Models: A Surveycited by 203
  2. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  3. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  4. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  5. 2025Time-R1: Post-Training Large Vision Language Model for Temporal Video Groundingcited by 138
  6. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  7. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  8. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 124
  9. 2024Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoningcited by 122
  10. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
  11. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  12. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 106
  13. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  14. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  15. 2025TerraMind: Large-Scale Generative Multimodality for Earth Observationcited by 101
  16. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 99
  17. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videoscited by 95
  18. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 90
  19. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
  20. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 87
  21. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 87
  22. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  23. 2024HawkEye: Training Video-Text LLMs for Grounding Text in Videoscited by 83
  24. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  25. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 78
  26. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Modelscited by 74
  27. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understandingcited by 72
  28. 2025Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMscited by 72
  29. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 71
  30. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 69
  31. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 68
  32. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoningcited by 66
  33. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  34. 2024VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selectioncited by 64
  35. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 60
  36. 2024Online Video Understanding: OVBench and VideoChat-Onlinecited by 58
  37. 2025LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Callingcited by 58
  38. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 57

50 most cited of 399 All 399 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025DATE: Dynamic Absolute Time Enhancement for Long Video Understanding
  2. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answering
  3. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding
  4. 2025Seeing the Forest and the Trees: Query-Aware Tokenizer for Long-Video Multimodal Language Models
  5. 2024Temporal Grounding of Activities using Multimodal Large Language Models
  6. 2026Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events
  7. 2025Iterative Zoom-In: Temporal Interval Exploration for Long Video Understanding
  8. 2025Video-QTR: Query-Driven Temporal Reasoning Framework for Lightweight Video Understanding