mlsanityData to 7 Oct 2026

Elysium: Exploring Object-level Perception in Videos via MLLM

arXiv 2403.16558 ↗First posted 25 Mar 2024cs.CVcited by 48

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Multi-modal Large Language Models (MLLMs) have demonstrated their ability to perceive objects in still images, but their application in video-related tasks, such as object tracking, remains understudied. This lack of exploration is primarily due to two key challenges. Firstly, extensive pretraining on large-scale video datasets is required to equip MLLMs with the capability to perceive objects across multiple frames and understand inter-frame relationships. Secondly, processing a large number of frames within the context window of Large Language Models (LLMs) can impose a significant computational burden. To address the first challenge, we introduce ElysiumTrack-1M, a large-scale video dataset supported for three tasks: Single Object Tracking (SOT), Referring Single Object Tracking (RSOT), and Video Referring Expression Generation (Video-REG). ElysiumTrack-1M contains 1.27 million annotated video frames with corresponding object boxes and descriptions. Leveraging this dataset, we conduct training of MLLMs and propose a token-compression model T-Selector to tackle the second challenge. Our proposed approach, Elysium: Exploring Object-level Perception in Videos via MLLM, is an end-to-end trainable MLLM that attempts to conduct object-level tasks in videos without requiring any additional plug-in or expert models. All codes and datasets are available at https://github.com/Hon-Wong/Elysium.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 80 references that are in the map. First posted on arXiv 25 Mar 2024.

Builds on

55 of 80 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding4 Dec
  2. 2023PG-Video-LLaVA: Pixel Grounding Large Video-Language Models22 Nov
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  11. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  12. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
38 more
  1. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  2. 2017Simple Online and Realtime Tracking with a Deep Association Metriccited by 908
  3. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  4. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  5. 2016Simple Online and Realtime Trackingcited by 862
  6. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  7. 2017Dense-Captioning Events in Videoscited by 789
  8. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  9. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  10. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  11. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  12. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  13. 2016Fully-Convolutional Siamese Networks for Object Trackingcited by 566
  14. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  15. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  16. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  17. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  18. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 471
  19. 2018SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networkscited by 445
  20. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  21. 2018LaSOT: A High-quality Benchmark for Large-scale Single Object Trackingcited by 424
  22. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 381
  23. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  24. 2019Learning Discriminative Model Prediction for Trackingcited by 344
  25. 2018TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wildcited by 342
  26. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  27. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  28. 2022MixFormer: End-to-End Tracking with Iterative Mixed Attentioncited by 278
  29. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  30. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  31. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  32. 2019SiamFC++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelinescited by 177
  33. 2018Video Object Segmentation with Language Referring Expressionscited by 161
  34. 2015Exploring Models and Data for Image Question Answeringcited by 139
  35. 2018Actor and Action Video Segmentation from a Sentencecited by 105
  36. 2020Graph Attention Trackingcited by 70
  37. 2020Tracking by Instance Detection: A Meta-Learning Approachcited by 61
  38. 2018Real-Time MDNetcited by 39

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 22; 2026: 16.

48 papers.

Sort:

By area:

  1. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  2. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  3. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  4. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  5. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 80
  6. 2024VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMcited by 71
  7. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 57
  8. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpuscited by 55
  9. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
  10. 2025Vision as LoRAcited by 37
  11. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLMcited by 31
  12. 20253D Aware Region Prompted Vision Language Modelcited by 31
36 more
  1. 2025SpaceVLLM: Endowing Multimodal Large Language Model with Spatio-Temporal Video Grounding Capabilitycited by 28
  2. 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularitycited by 21
  3. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMscited by 18
  4. 2025Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Markscited by 17
  5. 2024VideoOrion: Tokenizing Object Dynamics in Videoscited by 12
  6. 2024VideoCogQA: A Controllable Benchmark for Evaluating Cognitive Abilities in Video-Language Modelscited by 10
  7. 2024How Texts Help? A Fine-grained Evaluation to Reveal the Role of Language in Vision-Language Trackingcited by 10
  8. 2025Unifying Specialized Visual Encoders for Video Language Modelscited by 10
  9. 2025iMOVE: Instance-Motion-Aware Video Understandingcited by 10
  10. 2026SAMTok: Representing Any Mask with Two Wordscited by 10
  11. 2024HumanVLM: Foundation for Human-Scene Vision-Language Modelcited by 8
  12. 2024INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuningcited by 8
  13. 2025CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videoscited by 8
  14. 2025Think Before You Segment: High-Quality Reasoning Segmentation with GPT Chain of Thoughtscited by 7
  15. 2025IntentVCNet: Bridging Spatio-Temporal Gaps for Intention-Oriented Controllable Video Captioningcited by 7
  16. 2025EVE: Towards End-to-End Video Subtitle Extraction with Vision-Language Modelscited by 6
  17. 2025Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Datacited by 6
  18. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  19. 2025ChineseVideoBench: Benchmarking Multi-modal Large Models for Chinese Video Question Answeringcited by 4
  20. 2026Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Groundingcited by 3
  21. 2025SUMMA: A Multimodal Large Language Model for Advertisement Summarizationcited by 2
  22. 2026Children's Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMscited by 2
  23. 2026From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Groundingcited by 2
  24. 2026TraceVision: Trajectory-Aware Vision-Language Model for Human-Like Spatial Understandingcited by 1
  25. 2026Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answeringcited by 1
  26. 2026See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understandingcited by 1
  27. 2026Credit the Right Box: Marginal Contribution Assignment for Structured Visual Perceptioncited by 1
  28. 2026Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMscited by 1
  29. 2025REVEAL: Relation-based Video Representation Learning for Video-Question-Answeringcited by 0
  30. 2025Harnessing Object Grounding for Time-Sensitive Video Understandingcited by 0
  31. 2026STORM: End-to-End Referring Multi-Object Tracking in Videoscited by 0
  32. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  33. 2026Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verificationcited by 0
  34. 2026Models as Tools: An Agentic Coordination Framework for Unified Multimodal Visual Trackingcited by 0
  35. 2026Video-FLAIR: Not Whether to Reason, But Howcited by 0
  36. 2026VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling
  2. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding
  3. 2024Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models
  4. 2026Benchmarking Visual State Tracking in Multimodal Video Understanding
  5. 2024MVP-Bench: Can Large Vision--Language Models Conduct Multi-level Visual Perception Like Humans?
  6. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
  7. 2024HourVideo: 1-Hour Video-Language Understanding
  8. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding