mlsanityData to 7 Oct 2026

Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding

arXiv 2501.07888 ↗First posted 14 Jan 2025cs.CV cs.AIcited by 80

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

We introduce Tarsier2, a state-of-the-art large vision-language model (LVLM) designed for generating detailed and accurate video descriptions, while also exhibiting superior general video understanding capabilities. Tarsier2 achieves significant advancements through three key upgrades: (1) Scaling pre-training data from 11M to 40M video-text pairs, enriching both volume and diversity; (2) Performing fine-grained temporal alignment during supervised fine-tuning; (3) Using model-based sampling to automatically construct preference data and applying DPO training for optimization. Extensive experiments show that Tarsier2-7B consistently outperforms leading proprietary models, including GPT-4o and Gemini 1.5 Pro, in detailed video description tasks. On the DREAM-1K benchmark, Tarsier2-7B improves F1 by 2.8% over GPT-4o and 5.8% over Gemini-1.5-Pro. In human side-by-side evaluations, Tarsier2-7B shows a +8.6% performance advantage over GPT-4o and +24.9% over Gemini-1.5-Pro. Tarsier2-7B also sets new state-of-the-art results across 15 public benchmarks, spanning tasks such as video question-answering, video grounding, hallucination test, and embodied question-answering, demonstrating its versatility as a robust generalist vision-language model.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 130 references that are in the map. First posted on arXiv 14 Jan 2025.

Builds on

51 of 130 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024StoryTeller: Improving Long Video Description through Global Audio-Visual Character Identification11 Nov
  2. 2024Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videos3 Oct
  3. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input28 Aug
  4. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding22 Jul
  5. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding20 Jun
  6. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  7. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning25 Apr
  8. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens4 Apr
  9. 2024Elysium: Exploring Object-level Perception in Videos via MLLM25 Mar
  10. 2023AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answering25 Nov
  11. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  9. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  10. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
27 more
  1. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  2. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  3. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  4. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  5. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  6. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  7. 2019Abductive Commonsense Reasoningcited by 300
  8. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  9. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  10. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  11. 2024Pixtral 12Bcited by 174
  12. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  13. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 136
  14. 2018Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videoscited by 127
  15. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 123
  16. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Roboticscited by 116
  17. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 102
  18. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 96
  19. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89
  20. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 88
  21. 2024VTG-LLM: Integrating Timestamp Knowledge into Video LLMs for Enhanced Video Temporal Groundingcited by 85
  22. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 71
  23. 2022VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captionerscited by 69
  24. 2024EventHallusion: Diagnosing Event Hallucinations in Video LLMscited by 55
  25. 2024Lost in Time: A New Temporal Benchmark for VideoLLMscited by 36
  26. 2020RareAct: A video dataset of unusual interactionscited by 23
  27. 2023An overview on the evaluated video retrieval tasks at TRECVID 2022cited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 43; 2026: 37.

80 papers.

Sort:

By area:

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  2. 2025Seedance 1.0: Exploring the Boundaries of Video Generation Modelscited by 247
  3. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 90
  4. 2025RELIC: Interactive Video World Model with Long-Horizon Memorycited by 89
  5. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
  6. 2025LongCat-Video Technical Reportcited by 73
  7. 2025Kwai Keye-VL 1.5 Technical Reportcited by 68
  8. 2025Goku: Flow Based Video Generative Foundation Modelscited by 67
  9. 2025Towards Understanding Camera Motions in Any Videocited by 62
  10. 2025Waver: Wave Your Way to Lifelike Video Generationcited by 52
  11. 2025Kwai Keye-VL Technical Reportcited by 47
  12. 2025InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generationcited by 43
38 more
  1. 2025OneStory: Coherent Multi-Shot Video Generation with Adaptive Memorycited by 31
  2. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 31
  3. 2025VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinkingcited by 27
  4. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  5. 2025AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestrationcited by 26
  6. 2025FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understandingcited by 24
  7. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  8. 2025Bidirectional Sparse Attention for Faster Video Diffusion Trainingcited by 18
  9. 2025MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learningcited by 18
  10. 2025WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perceptioncited by 14
  11. 2026Thinking in Streaming Videocited by 14
  12. 2025JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluationcited by 13
  13. 2026DiaDem: Advancing Dialogue Descriptions in Audiovisual Video Captioning for Multimodal Large Language Modelscited by 12
  14. 2025MotionSight: Boosting Fine-Grained Motion Understanding in Multimodal LLMscited by 11
  15. 2025OwlCap: Harmonizing Motion-Detail for Video Captioning via HMD-270K and Caption Set Equivalence Rewardcited by 11
  16. 2025VidBridge-R1: Bridging QA and Captioning for RL-based Video Understanding Models with Intermediate Proxy Taskscited by 10
  17. 2025UI2V-Bench: An Understanding-based Image-to-video Generation Benchmarkcited by 10
  18. 2026Egocentric World Model for Photorealistic Hand-Object Interaction Synthesiscited by 10
  19. 2026MV-S2V: Multi-View Subject-Consistent Video Generationcited by 9
  20. 2025VRR-QA: Visual Relational Reasoning in Videos Beyond Explicit Cuescited by 8
  21. 2026Coarse-to-Real: Generative Rendering for Populated Dynamic Scenescited by 8
  22. 2026TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captionscited by 8
  23. 2025IF-VidCap: Can Video Caption Models Follow Instructions?cited by 7
  24. 2026Towards Universal Video MLLMs with Attribute-Structured and Quality-Verified Instructionscited by 7
  25. 2026VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understandingcited by 6
  26. 2025Gather and Trace: Rethinking Video TextVQA from an Instance-oriented Perspectivecited by 5
  27. 2025CETCAM: Camera-Controllable Video Generation via Consistent and Extensible Tokenizationcited by 4
  28. 2026Syn4D: A Multiview Synthetic 4D Datasetcited by 4
  29. 2025TEMPLE: Incentivizing Temporal Understanding of Video Large Language Models via Progressive Pre-SFT Alignmentcited by 3
  30. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  31. 2025NeMo: Needle in a Montage for Video-Language Understandingcited by 3
  32. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 3
  33. 2026CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioningcited by 3
  34. 2026Claim-Level Rubric Rewards for Video Caption Reinforcement Learningcited by 3
  35. 2025Measure Twice, Cut Once: A Semantic-Oriented Approach to Video Temporal Localization with Video LLMscited by 2
  36. 2025Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMscited by 2
  37. 2025Does Hearing Help Seeing? Investigating Audio-Video Joint Denoising for Video Generationcited by 2
  38. 2026KPM-Bench: A Kinematic Parsing Motion Benchmark for Fine-grained Motion-centric Video Understandingcited by 2

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models
  2. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models
  3. 2025FrameOracle: Learning What to See and How Much to See in Videos
  4. 2025PEVLM: Parallel Encoding for Vision-Language Models
  5. 2026PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models
  6. 2025Unifying Specialized Visual Encoders for Video Language Models
  7. 2025V2P-Bench: Evaluating Video-Language Understanding with Visual Prompts for Better Human-Model Interaction
  8. 2025Temporal-Oriented Recipe for Transferring Large Vision-Language Model to Video Understanding