mlsanityData to 7 Oct 2026

VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM

arXiv 2501.00599 ↗First posted 31 Dec 2024cs.CV cs.AI cs.LGcited by 71

Not on the map. Why?

Cited by 25 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Video Large Language Models (Video LLMs) have recently exhibited remarkable capabilities in general video understanding. However, they mainly focus on holistic comprehension and struggle with capturing fine-grained spatial and temporal details. Besides, the lack of high-quality object-level video instruction data and a comprehensive benchmark further hinders their advancements. To tackle these challenges, we introduce the VideoRefer Suite to empower Video LLM for finer-level spatial-temporal video understanding, i.e., enabling perception and reasoning on any objects throughout the video. Specially, we thoroughly develop VideoRefer Suite across three essential aspects: dataset, model, and benchmark. Firstly, we introduce a multi-agent data engine to meticulously curate a large-scale, high-quality object-level video instruction dataset, termed VideoRefer-700K. Next, we present the VideoRefer model, which equips a versatile spatial-temporal object encoder to capture precise regional and sequential representations. Finally, we meticulously create a VideoRefer-Bench to comprehensively assess the spatial-temporal understanding capability of a Video LLM, evaluating it across various aspects. Extensive experiments and analyses demonstrate that our VideoRefer model not only achieves promising performance on video referring benchmarks but also facilitates general video understanding capabilities.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 48 of its 59 references that are in the map. First posted on arXiv 31 Dec 2024.

Builds on

48 of 59 references.

References · 48

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  7. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  9. 2024Qwen2 Technical Reportcited by 2,076
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  12. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
36 more
  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  3. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  5. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  7. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  8. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  9. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  10. 2024Long Context Transfer from Language to Visioncited by 502
  11. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  12. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  13. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  14. 2023Segment Anything in High Qualitycited by 373
  15. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  16. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  17. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  18. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 225
  19. 2023Video Understanding with Large Language Models: A Surveycited by 203
  20. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  21. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  22. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158
  23. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  24. 2020Human-centric Spatio-Temporal Video Grounding With Visual Transformerscited by 83
  25. 2024RegionGPT: Towards Region Understanding Vision Language Modelcited by 71
  26. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuningcited by 65
  27. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Modelscited by 59
  28. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  29. 2024Elysium: Exploring Object-level Perception in Videos via MLLMcited by 48
  30. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMscited by 47
  31. 2023Merlin:Empowering Multimodal LLMs with Foresight Mindscited by 46
  32. 2024Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referringcited by 29
  33. 2024Artemis: Towards Referential Understanding in Complex Videoscited by 25
  34. 2024ChatterBox: Multi-round Multimodal Referring and Groundingcited by 24
  35. 2024HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Modelscited by 19
  36. 2024SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 32; 2026: 39.

71 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  3. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
  4. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 43
  5. 2025UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoningcited by 27
  6. 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularitycited by 21
  7. 2025EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?cited by 20
  8. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMscited by 18
  9. 2025DynamicVL: Benchmarking Multimodal Large Language Models for Dynamic City Understandingcited by 17
  10. 2025Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMscited by 16
  11. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 15
  12. 2025VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Modelscited by 14
38 more
  1. 2025Object-centric Video Question Answering with Visual Grounding and Referringcited by 12
  2. 2025RynnEC: Bringing MLLMs into Embodied Worldcited by 11
  3. 2025WeDetect: Fast Open-Vocabulary Object Detection as Retrievalcited by 11
  4. 2025iMOVE: Instance-Motion-Aware Video Understandingcited by 10
  5. 2025V2P-Bench: Evaluating Video-Language Understanding with Visual Prompts for Better Human-Model Interactioncited by 10
  6. 2025UI2V-Bench: An Understanding-based Image-to-video Generation Benchmarkcited by 10
  7. 2025Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Taskcited by 10
  8. 2026SAMTok: Representing Any Mask with Two Wordscited by 10
  9. 2026OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysiscited by 9
  10. 2025SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Modelscited by 8
  11. 2026Unified Personalized Understanding, Generating and Editingcited by 8
  12. 2026AgentVLN: Towards Agentic Vision-and-Language Navigationcited by 8
  13. 2025FrameOracle: Learning What to See and How Much to See in Videoscited by 7
  14. 2025Video-LevelGauge: Investigating Contextual Positional Bias in Large Video Language Modelscited by 6
  15. 2025Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Datacited by 6
  16. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  17. 2026InstructSAM: Segment Any Instance with Any Instructionscited by 6
  18. 2026VideoLoom: A Video Large Language Model for Joint Spatial-Temporal Understandingcited by 5
  19. 2025A Unified Multi-Agent Framework for Universal Multimodal Understanding and Generationcited by 4
  20. 2026MME-CoF-Pro: Evaluating Reasoning Coherence in Video Generative Models with Text and Visual Hintscited by 4
  21. 2026Make Geometry Matter for Spatial Reasoningcited by 4
  22. 2026VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policiescited by 4
  23. 2025FocusedAD: Character-centric Movie Audio Descriptioncited by 3
  24. 2025MUVR: A Multi-Modal Untrimmed Video Retrieval Benchmark with Multi-Level Visual Correspondencecited by 3
  25. 2026Are VLMs Lost Between Sky and Space? LinkS2Bench for UAV-Satellite Dynamic Cross-View Spatial Intelligencecited by 3
  26. 2025UFVideo: Towards Unified Fine-Grained Video Cooperative Understanding with Large Language Modelscited by 2
  27. 2026Next-Gen CAPTCHAs: Leveraging the Cognitive Gap for Scalable and Diverse GUI-Agent Defensecited by 2
  28. 2025Fast Thinking for Large Language Modelscited by 1
  29. 2025Bridging the Gap Between Multimodal Foundation Models and World Modelscited by 1
  30. 2025EBind: a practical approach to space bindingcited by 1
  31. 2025HFS: Holistic Query-Aware Frame Selection for Efficient Video Understandingcited by 1
  32. 2025Temporal Visual Semantics-Induced Human Motion Understanding with Large Language Modelscited by 1
  33. 2026MACD: Model-Aware Contrastive Decoding via Counterfactual Datacited by 1
  34. 2026Enhancing Post-Training Quantization via Future Activation Awarenesscited by 1
  35. 2026TraceVision: Trajectory-Aware Vision-Language Model for Human-Like Spatial Understandingcited by 1
  36. 2026Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answeringcited by 1
  37. 2026Chain-of-Glimpse: Search-Guided Progressive Object-Grounded Reasoning for Video Understandingcited by 1
  38. 2026See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understandingcited by 1

50 most cited of 71 All 71 (JSON)