mlsanityData to 7 Oct 2026

HawkEye: Training Video-Text LLMs for Grounding Text in Videos

arXiv 2403.10228 ↗First posted 15 Mar 2024cs.CV cs.AI cs.CLcited by 83

Not on the map. Why?

Cited by 28 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Video-text Large Language Models (video-text LLMs) have shown remarkable performance in answering questions and holding conversations on simple videos. However, they perform almost the same as random on grounding text queries in long and complicated videos, having little ability to understand and reason about temporal information, which is the most fundamental difference between videos and images. In this paper, we propose HawkEye, one of the first video-text LLMs that can perform temporal video grounding in a fully text-to-text manner. To collect training data that is applicable for temporal video grounding, we construct InternVid-G, a large-scale video-text corpus with segment-level captions and negative spans, with which we introduce two new time-aware training objectives to video-text LLMs. We also propose a coarse-grained method of representing segments in videos, which is more robust and easier for LLMs to learn and follow than other alternatives. Extensive experiments show that HawkEye is better at temporal video grounding and comparable on other video-text tasks with existing video-text LLMs, which verifies its superior video-text multi-modal understanding abilities.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 41 of its 45 references that are in the map. First posted on arXiv 15 Mar 2024.

Builds on

41 of 45 references.

References · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  8. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  9. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  11. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  12. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
29 more
  1. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  2. 2017Dense-Captioning Events in Videoscited by 789
  3. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  4. 2017Localizing Moments in Video with Natural Languagecited by 660
  5. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  6. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  7. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  8. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  9. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  10. 2019Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Languagecited by 283
  11. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  12. 2020Span-based Localizing Network for Natural Language Video Localizationcited by 234
  13. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  14. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  15. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  16. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  17. 2023Query-Dependent Video Representation for Moment Retrieval and Highlight Detectioncited by 166
  18. 2018To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regressioncited by 157
  19. 2018MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustmentcited by 145
  20. 2019Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videoscited by 135
  21. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  22. 2021Interventional Video Grounding with Dual Contrastive Learningcited by 94
  23. 2019Weakly Supervised Video Moment Retrieval From Text Queriescited by 85
  24. 2021QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queriescited by 72
  25. 2019Weakly-Supervised Video Moment Retrieval via Semantic Completion Networkcited by 69
  26. 2018Weakly Supervised Dense Event Captioning in Videoscited by 58
  27. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  28. 2021Adaptive Proposal Generation Network for Temporal Sentence Localization in Videoscited by 42
  29. 2022Reducing the Vision and Language Bias for Temporal Sentence Groundingcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 39; 2026: 28.

83 papers.

Sort:

By area:

  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  2. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  3. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  4. 2025Time-R1: Post-Training Large Vision Language Model for Temporal Video Groundingcited by 138
  5. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 99
  6. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 88
  7. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
  8. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 87
  9. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 68
  10. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  11. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 57
  12. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 57
38 more
  1. 2024ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videoscited by 51
  2. 2024VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Formatcited by 44
  3. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
  4. 2026Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillationcited by 32
  5. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  6. 2025SpaceVLLM: Endowing Multimodal Large Language Model with Spatio-Temporal Video Grounding Capabilitycited by 28
  7. 2025Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiencycited by 25
  8. 2025LiveStar: Live Streaming Assistant for Real-World Online Video Understandingcited by 22
  9. 2025Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigationcited by 20
  10. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Modelcited by 19
  11. 2024On the Consistency of Video Large Language Models in Temporal Comprehensioncited by 18
  12. 2024ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videoscited by 18
  13. 2024TimeRefine: Temporal Grounding with Time Refining Video LLMcited by 15
  14. 2025TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMscited by 15
  15. 2025TimeExpert: An Expert-Guided Video LLM for Video Temporal Groundingcited by 14
  16. 2025Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generationcited by 13
  17. 2025TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Actioncited by 13
  18. 2024Seq2Time: Sequential Knowledge Transfer for Video LLM Temporal Groundingcited by 12
  19. 2025Cross-modal Causal Relation Alignment for Video Question Groundingcited by 12
  20. 2025Enrich and Detect: Video Temporal Grounding with Multimodal LLMscited by 12
  21. 2025ZoomV: Temporal Zoom-in for Efficient Long Video Understandingcited by 10
  22. 2025Tempo-R0: A Video-MLLM for Temporal Video Grounding through Efficient Temporal Sensing Reinforcement Learningcited by 9
  23. 2025VideoExpert: Augmented LLM for Temporal-Sensitive Video Understandingcited by 8
  24. 2026CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modelingcited by 8
  25. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
  26. 2025VideoTG-R1: Boosting Video Temporal Grounding via Curriculum Reinforcement Learning on Reflected Boundary Annotationscited by 7
  27. 2025TempR1: Improving Temporal Understanding of MLLMs via Temporal-Aware Multi-Task Reinforcement Learningcited by 7
  28. 2025VITED: Video Temporal Evidence Distillationcited by 5
  29. 2025TAR: Temporal Anchor-Constrained Reasoning for Video Temporal Groundingcited by 5
  30. 2025TimeScope: Towards Task-Oriented Temporal Grounding In Long Videoscited by 5
  31. 2025ChronusOmni: Improving Time Awareness of Omni Large Language Modelscited by 5
  32. 2026VideoLoom: A Video Large Language Model for Joint Spatial-Temporal Understandingcited by 5
  33. 2025MUPA: Towards Multi-Path Agentic Reasoning for Grounded Video Question Answeringcited by 4
  34. 2025MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understandingcited by 4
  35. 2025EgoExo-Con: Exploring View-Invariant Video Temporal Understandingcited by 4
  36. 2025Factorized Learning for Temporally Grounded Video-Language Modelscited by 4
  37. 2026ArrowGEV: Grounding Events in Video via Learning the Arrow of Timecited by 4
  38. 2026OmniVTG: A Large-Scale Dataset and Training Paradigm for Open-World Video Temporal Groundingcited by 4

50 most cited of 83 All 83 (JSON)