mlsanityData to 14 Oct 2026

TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Ability

arXiv 2411.18211 ↗First posted 27 Nov 2024cs.CV cs.AIcited by 70

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Rapid development of large language models (LLMs) has significantly advanced multimodal large language models (LMMs), particularly in vision-language tasks. However, existing video-language models often overlook precise temporal localization and struggle with videos of varying lengths. We introduce TimeMarker, a versatile Video-LLM designed for high-quality dialogue based on video content, emphasizing temporal localization. TimeMarker integrates Temporal Separator Tokens to enhance temporal awareness, accurately marking specific moments within videos. It employs the AnyLength mechanism for dynamic frame sampling and adaptive token merging, enabling effective handling of both short and long videos. Additionally, TimeMarker utilizes diverse datasets, including further transformed temporal-related video QA datasets, to bolster its temporal understanding capabilities. Image and interleaved data are also employed to further enhance the model's semantic perception ability. Evaluations demonstrate that TimeMarker achieves state-of-the-art performance across multiple benchmarks, excelling in both short and long video categories. Our project page is at urlhttps://github.com/TimeMarker-LLM/TimeMarker/.

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 72 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

60 of 72 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding22 Oct
  2. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding22 Sep
  3. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture4 Sep
  4. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input28 Aug
  5. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding22 Jul
  6. 2024Long Context Transfer from Language to Vision24 Jun
  7. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding20 Jun
  8. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoning17 Jun
  9. 2024LVBench: An Extreme Long Video Understanding Benchmark12 Jun
  10. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  11. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning25 Apr
  12. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding8 Apr
4 more
  1. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding4 Dec
  2. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
32 more
  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  2. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  3. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  5. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  6. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  7. 2017Dense-Captioning Events in Videoscited by 793
  8. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  9. 2017TALL: Temporal Activity Localization via Language Querycited by 584
  10. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  11. 2024InternLM2 Technical Reportcited by 397
  12. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  13. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  14. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 292
  15. 2019Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Languagecited by 285
  16. 2017Temporal Action Detection with Structured Segment Networkscited by 267
  17. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 255
  18. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
  19. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  20. 2023UniVTG: Towards Unified Video-Language Temporal Groundingcited by 190
  21. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  22. 2018To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regressioncited by 157
  23. 2019Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videoscited by 135
  24. 2024Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoningcited by 122
  25. 2021Negative Sample Matters: A Renaissance of Metric Learning for Temporal Groundingcited by 90
  26. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  27. 2019Less is More: Learning Highlight Detection from Video Durationcited by 60
  28. 2018Weakly Supervised Dense Event Captioning in Videoscited by 58
  29. 2024From LLM to NMT: Advancing Low-Resource Machine Translation with Claudecited by 58
  30. 2024ChatVTG: Video Temporal Grounding via Chat with Video Dialogue Large Language Modelscited by 54
  31. 2021Deep Learning-based Action Detection in Untrimmed Videos: A Surveycited by 32
  32. 2022CCMB: A Large-scale Chinese Cross-modal Benchmarkcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 35; 2026: 30.

70 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,793
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  3. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 165
  4. 2025TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMscited by 50
  5. 2025Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understandingcited by 49
  6. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  7. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 43
  8. 2026Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillationcited by 36
  9. 2024ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understandingcited by 33
  10. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivingcited by 30
  11. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  12. 2025Temporal Preference Optimization for Long-Form Video Understandingcited by 29
38 more
  1. 2025Universal Video Temporal Grounding with Generative Multi-modal Large Language Modelscited by 27
  2. 2025LongCat-Flash-Omni Technical Reportcited by 26
  3. 2025MR. Video: "MapReduce" is the Principle for Long Video Understandingcited by 23
  4. 2025VideoLucy: Deep Memory Backtracking for Long Video Understandingcited by 22
  5. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Modelcited by 20
  6. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 18
  7. 2025ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understandingcited by 16
  8. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 16
  9. 2025TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Actioncited by 14
  10. 2025Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generationcited by 13
  11. 2026Lance: Unified Multimodal Modeling by Multi-Task Synergycited by 12
  12. 2026CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modelingcited by 11
  13. 2026MOSS-Audio Technical Reportcited by 11
  14. 2025iMOVE: Instance-Motion-Aware Video Understandingcited by 10
  15. 2026Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanismcited by 9
  16. 2025VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignmentcited by 8
  17. 2026Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisitioncited by 8
  18. 2025DisTime: Distribution-based Time Representation for Video Large Language Modelscited by 7
  19. 2025UniComp: Rethinking Video Compression Through Informational Uniquenesscited by 7
  20. 2026VideoTemp-o3: Harmonizing Temporal Grounding and Video Understanding in Agentic Thinking-with-Videoscited by 7
  21. 2026LongVideo-R1: Smart Navigation for Low-cost Long Video Understandingcited by 7
  22. 2025LVC: A Lightweight Compression Framework for Enhancing VLMs in Long Video Understandingcited by 6
  23. 2025Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoderscited by 6
  24. 2025ChronusOmni: Improving Time Awareness of Omni Large Language Modelscited by 6
  25. 2025OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactionscited by 5
  26. 2025RAVEN: Robust Advertisement Video Violation Temporal Grounding via Reinforcement Reasoningcited by 5
  27. 2025VideoPerceiver: Enhancing Fine-Grained Temporal Perception in Video Multimodal Large Language Modelscited by 5
  28. 2026TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Groundingcited by 5
  29. 2026One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understandingcited by 5
  30. 2025Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMscited by 4
  31. 2025MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understandingcited by 4
  32. 2025RAVEN++: Pinpointing Fine-Grained Violations in Advertisement Videos with Active Reinforcement Reasoningcited by 4
  33. 2026MOSS Transcribe Diarize Technical Reportcited by 4
  34. 2024Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignmentcited by 3
  35. 2025TEMPLE: Incentivizing Temporal Understanding of Video Large Language Models via Progressive Pre-SFT Alignmentcited by 3
  36. 2026Query-Conditioned Evidential Keyframe Sampling for MLLM-Based Long-Form Video Understandingcited by 3
  37. 2026OmniReasoner: Thinking with Long Audio-Video via Native Tool Usecited by 3
  38. 2025From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Modelscited by 2

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026TOC-Bench: A Temporal Object Consistency Benchmark for Video Large Language Models
  2. 2025VideoExpert: Augmented LLM for Temporal-Sensitive Video Understanding
  3. 2024Do Language Models Understand Time?
  4. 2024Chrono: A Simple Blueprint for Representing Time in MLLMs
  5. 2024Temporal Grounding of Activities using Multimodal Large Language Models
  6. 2025Harnessing Object Grounding for Time-Sensitive Video Understanding
  7. 2025DATE: Dynamic Absolute Time Enhancement for Long Video Understanding
  8. 2024LITA: Language Instructed Temporal-Localization Assistant