mlsanityData to 14 Oct 2026

Artemis: Towards Referential Understanding in Complex Videos

arXiv 2406.00258 ↗First posted 1 Jun 2024cs.CV cs.AIcited by 25

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Videos carry rich visual information including object description, action, interaction, etc., but the existing multimodal large language models (MLLMs) fell short in referential understanding scenarios such as video-based referring. In this paper, we present Artemis, an MLLM that pushes video-based referential understanding to a finer level. Given a video, Artemis receives a natural-language question with a bounding box in any video frame and describes the referred target in the entire video. The key to achieving this goal lies in extracting compact, target-specific video features, where we set a solid baseline by tracking and selecting spatiotemporal features from the video. We train Artemis on the newly established VideoRef45K dataset with 45K video-QA pairs and design a computationally efficient, three-stage training procedure. Results are promising both quantitatively and qualitatively. Additionally, we show that model can be integrated with video grounding and text summarization tools to understand more complex scenarios. Code and data are available at https://github.com/qiujihao19/Artemis.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 66 references that are in the map. First posted on arXiv 1 Jun 2024.

Builds on

51 of 66 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2023Segment Anythingcited by 8,234
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2017Mask R-CNNcited by 7,277
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  11. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  12. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  13. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  14. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  15. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  16. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  17. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  18. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  19. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  20. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  21. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  22. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  23. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 475
  24. 2018LaSOT: A High-quality Benchmark for Large-scale Single Object Trackingcited by 429
  25. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  26. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  27. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  28. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 228
  29. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  30. 2018Actor and Action Video Segmentation from a Sentencecited by 106
  31. 2020Human-centric Spatio-Temporal Video Grounding With Visual Transformerscited by 85
  32. 2019Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Videocited by 60
  33. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Modelscited by 59
  34. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMscited by 47
  35. 2022Fast-iTPN: Integrally Pre-Trained Transformer Pyramid Network with Token Migrationcited by 24
  36. 2024ChatterBox: Multi-round Multimodal Referring and Groundingcited by 24
  37. 2021Semantic-Aware Generation for Self-Supervised Visual Representation Learningcited by 11
  38. 2023Tracking Anything in High Qualitycited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 11; 2026: 9.

25 papers.

Sort:

By area:

  1. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 167
  2. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  3. 2024VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMcited by 72
  4. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 51
  5. 20253D Aware Region Prompted Vision Language Modelcited by 33
  6. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understandingcited by 30
  7. 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularitycited by 21
  8. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMscited by 18
  9. 2025Object-centric Video Question Answering with Visual Grounding and Referringcited by 13
  10. 2024VideoOrion: Tokenizing Object Dynamics in Videoscited by 12
  11. 2024Personalized Large Vision-Language Modelscited by 12
  12. 2025Long-Video Audio Synthesis with Multi-Agent Collaborationcited by 8
13 more
  1. 2025SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Modelscited by 8
  2. 2026LongVideo-R1: Smart Navigation for Low-cost Long Video Understandingcited by 7
  3. 2025Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Datacited by 6
  4. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  5. 2026TRACE: Evidence Grounding-Guided Multi-Video Event Understanding and Claim Generationcited by 3
  6. 2024IQViC: In-context, Question Adaptive Vision Compressor for Long-term Video Understanding LMMscited by 2
  7. 2025UFVideo: Towards Unified Fine-Grained Video Cooperative Understanding with Large Language Modelscited by 2
  8. 2026TraceVision: Trajectory-Aware Vision-Language Model for Human-Like Spatial Understandingcited by 2
  9. 2026SPARROW: Learning Spatial Precision and Temporal Referential Consistency in Pixel-Grounded Video MLLMscited by 2
  10. 2026See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understandingcited by 2
  11. 2026Intend, Reflect, Refine: An Adaptive Multimodal Reflection Framework for Autonomous Drivingcited by 2
  12. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  13. 2026CheXGround: Anatomical Region Tokens for Grounded Longitudinal Chest X-ray Interpretationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding
  2. 2024HourVideo: 1-Hour Video-Language Understanding
  3. 2023Vamos: Versatile Action Models for Video Understanding
  4. 2026VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding
  5. 2025Toward Scalable Video Narration: A Training-free Approach Using Multimodal Large Language Models
  6. 2023VaQuitA: Enhancing Alignment in LLM-Assisted Video Understanding
  7. 2025Unleashing Hour-Scale Video Training for Long Video-Language Understanding
  8. 2025Video Panels for Long Video Understanding