mlsanityData to 7 Oct 2026

Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models

arXiv 2410.03290 ↗First posted 4 Oct 2024cs.CV cs.AIcited by 87

Not on the map. Why?

Cited by 33 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Video Large Language Models (Video-LLMs) have demonstrated remarkable capabilities in coarse-grained video understanding, however, they struggle with fine-grained temporal grounding. In this paper, we introduce Grounded-VideoLLM, a novel Video-LLM adept at perceiving and reasoning over specific video moments in a fine-grained manner. We identify that current Video-LLMs have limitations for fine-grained video understanding since they lack effective temporal modeling and timestamp representation. In light of this, we sharpen our model by incorporating (1) an additional temporal stream to encode the relationships between frames and (2) discrete temporal tokens enriched with specific time knowledge to represent timestamps. To optimize the training of Grounded-VideoLLM, we employ a multi-stage training scheme, beginning with simple video-captioning tasks and progressively introducing video temporal grounding tasks of increasing complexity. To further enhance Grounded-VideoLLM's temporal reasoning capability, we also curate a grounded VideoQA dataset by an automatic annotation pipeline. Extensive experiments demonstrate that Grounded-VideoLLM not only excels in fine-grained grounding tasks such as temporal sentence grounding, dense video captioning, and grounded VideoQA, but also shows great potential as a versatile video assistant for general video understanding.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 58 of its 68 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

58 of 68 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  11. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
38 more
  1. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  3. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  4. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  6. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  7. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  8. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  9. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  10. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  11. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  12. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  13. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  14. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  15. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  16. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  17. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  18. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  19. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  20. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  21. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  22. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  23. 2023Mathematical Capabilities of ChatGPTcited by 263
  24. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  25. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  26. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  27. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  28. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  29. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  30. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  31. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  32. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  33. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  34. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 124
  35. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 123
  36. 2024Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoningcited by 122
  37. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understandingcited by 113
  38. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 88

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 47; 2026: 35.

87 papers.

Sort:

By area:

  1. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  2. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  3. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 57
  4. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
  5. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 52
  6. 2025TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMscited by 46
  7. 2024VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Formatcited by 44
  8. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 43
  9. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
  10. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 38
  11. 2026Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillationcited by 32
  12. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 31
38 more
  1. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 31
  2. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  3. 2025Temporal Preference Optimization for Long-Form Video Understandingcited by 28
  4. 2025SpaceVLLM: Endowing Multimodal Large Language Model with Spatio-Temporal Video Grounding Capabilitycited by 28
  5. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 28
  6. 2025Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiencycited by 25
  7. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Modelcited by 19
  8. 2025Datasets and Recipes for Video Temporal Grounding via Reinforcement Learningcited by 18
  9. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 17
  10. 2025Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMscited by 16
  11. 2024TimeRefine: Temporal Grounding with Time Refining Video LLMcited by 15
  12. 2025TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMscited by 15
  13. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 15
  14. 2025Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning without 3D Annotationscited by 14
  15. 2025TimeExpert: An Expert-Guided Video LLM for Video Temporal Groundingcited by 14
  16. 2025Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generationcited by 13
  17. 2025TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Actioncited by 13
  18. 2025Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-grained Spatial-Temporal Groundingcited by 13
  19. 2024Seq2Time: Sequential Knowledge Transfer for Video LLM Temporal Groundingcited by 12
  20. 2025Enrich and Detect: Video Temporal Grounding with Multimodal LLMscited by 12
  21. 2025ZoomV: Temporal Zoom-in for Efficient Long Video Understandingcited by 10
  22. 2025Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Taskcited by 10
  23. 2025VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignmentcited by 8
  24. 2025TOGA: Temporally Grounded Open-Ended Video QA with Weak Supervisioncited by 8
  25. 2025Zoom-Zero: Reinforced Coarse-to-Fine Video Understanding via Temporal Zoom-incited by 8
  26. 2025Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Datacited by 6
  27. 2026VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understandingcited by 6
  28. 2025LLaFEA: Frame-Event Complementary Fusion for Fine-Grained Spatiotemporal Understanding in LMMscited by 5
  29. 2025MASR: Self-Reflective Reasoning through Multimodal Hierarchical Attention Focusing for Agent-based Video Understandingcited by 4
  30. 2025MUPA: Towards Multi-Path Agentic Reasoning for Grounded Video Question Answeringcited by 4
  31. 2025MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understandingcited by 4
  32. 2025Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Groundingcited by 4
  33. 2025Factorized Learning for Temporally Grounded Video-Language Modelscited by 4
  34. 2025Poisoning Prompt-Guided Sampling in Video Large Language Modelscited by 3
  35. 2026EvoGround: Self-Evolving Video Agents for Video Temporal Groundingcited by 3
  36. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 3
  37. 2025Towards Efficient and Robust Moment Retrieval System: A Unified Framework for Multi-Granularity Models and Temporal Rerankingcited by 2
  38. 2025Ask2Loc: Learning to Locate Instructional Visual Answers by Asking Questionscited by 2

50 most cited of 87 All 87 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Universal Video Temporal Grounding with Generative Multi-modal Large Language Models
  2. 2023VideoLLM: Modeling Video Sequence with Large Language Models
  3. 2025Harnessing Object Grounding for Time-Sensitive Video Understanding
  4. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning
  5. 2024ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos
  6. 2025Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding
  7. 2025How Important are Videos for Training Video LLMs?
  8. 2024Training-free Video Temporal Grounding using Large-scale Pre-trained Models