mlsanityData to 7 Oct 2026

Tarsier: Recipes for Training and Evaluating Large Video Description Models

arXiv 2407.00634 ↗First posted 30 Jun 2024cs.CV cs.LGcited by 140

Not on the map. Why?

Cited by 46 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generating fine-grained video descriptions is a fundamental challenge in video understanding. In this work, we introduce Tarsier, a family of large-scale video-language models designed to generate high-quality video descriptions. Tarsier employs CLIP-ViT to encode frames separately and then uses an LLM to model temporal relationships. Despite its simple architecture, we demonstrate that with a meticulously designed two-stage training procedure, the Tarsier models exhibit substantially stronger video description capabilities than any existing open-source model, showing a +51.4% advantage in human side-by-side evaluation over the strongest model. Additionally, they are comparable to state-of-the-art proprietary models, with a +12.3% advantage against GPT-4V and a -6.7% disadvantage against Gemini 1.5 Pro. When upgraded to Tarsier2 by building upon SigLIP and Qwen2-7B, it further improves significantly with a +4.8% advantage against GPT-4o. Besides video description, Tarsier proves to be a versatile generalist model, achieving new state-of-the-art results across nine public benchmarks, including multi-choice VQA, open-ended VQA, and zero-shot video captioning. Our second contribution is the introduction of a new benchmark -- DREAM-1K (https://tarsier-vlm.github.io/) for evaluating video description models, consisting of a new challenging dataset featuring videos from diverse sources and varying complexity, along with an automatic method specifically designed to assess the quality of fine-grained video descriptions. We make our models and evaluation benchmark publicly available at https://github.com/bytedance/tarsier.

Key facts. Cited by 140 papers in the map (data through 7 Oct 2026). Builds on 73 of its 90 references that are in the map. First posted on arXiv 30 Jun 2024.

Builds on

73 of 90 references.

References · 73

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  12. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
38 more
  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  2. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  3. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  4. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  5. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  6. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  7. 2021Video Swin Transformercited by 1,090
  8. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  10. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  11. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  12. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  13. 2017Dense-Captioning Events in Videoscited by 789
  14. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  15. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  16. 2017Localizing Moments in Video with Natural Languagecited by 660
  17. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  18. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  19. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  20. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  21. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  22. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  23. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  24. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  25. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  26. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  27. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  28. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  29. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  30. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  31. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  32. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  33. 2023Tracking Anything with Decoupled Video Segmentationcited by 197
  34. 2016Movie Descriptioncited by 180
  35. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  36. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  37. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  38. 2023Emu: Generative Pretraining in Multimodalitycited by 156

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 82; 2026: 41.

140 papers.

Sort:

By area:

  1. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  3. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  4. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 248
  5. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  6. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
  7. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
  8. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 87
  9. 2025WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generationcited by 87
  10. 2025T*: Re-thinking Temporal Search for Long-Form Video Understandingcited by 82
  11. 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generationcited by 77
  12. 2025Goku: Flow Based Video Generative Foundation Modelscited by 67
38 more
  1. 2024PruneVid: Visual Token Pruning for Efficient Video Large Language Modelscited by 64
  2. 2025Towards Understanding Camera Motions in Any Videocited by 62
  3. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 61
  4. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
  5. 2025NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamicscited by 40
  6. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 38
  7. 2024Lost in Time: A New Temporal Benchmark for VideoLLMscited by 36
  8. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformerscited by 35
  9. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  10. 2024MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Modelscited by 34
  11. 2025Unhackable Temporal Rewarding for Scalable Video MLLMscited by 33
  12. 2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inferencecited by 31
  13. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 31
  14. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  15. 2025A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairscited by 30
  16. 2025Seeing the Arrow of Time in Large Multimodal Modelscited by 28
  17. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 28
  18. 2025VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinkingcited by 27
  19. 2025FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understandingcited by 26
  20. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  21. 2025AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestrationcited by 26
  22. 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understandingcited by 26
  23. 2024PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Modelscited by 25
  24. 2024SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Modelscited by 24
  25. 2025FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understandingcited by 24
  26. 2025CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughnesscited by 23
  27. 2025BIMBA: Selective-Scan Compression for Long-Range Video Question Answeringcited by 23
  28. 2025VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generationcited by 22
  29. 2026Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddingscited by 22
  30. 2025EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?cited by 20
  31. 2024LinVT: Empower Your Image-level Large Language Model to Understand Videoscited by 19
  32. 2025Hybrid-Level Instruction Injection for Video Token Compression in Multi-modal Large Language Modelscited by 19
  33. 2024CaReBench: A Fine-Grained Benchmark for Video Captioning and Retrievalcited by 18
  34. 2025UGC-VideoCaptioner: An Omni UGC Video Detail Caption Model and New Benchmarkscited by 17
  35. 2025VideoMultiAgents: A Multi-Agent Framework for Video Question Answeringcited by 16
  36. 2025RoadSocial: A Diverse VideoQA Dataset and Benchmark for Road Event Understanding from Social Video Narrativescited by 15
  37. 2026AURA: Always-On Understanding and Real-Time Assistance via Video Streamscited by 15
  38. 2025LoVR: A Benchmark for Long Video Retrieval in Multimodal Contextscited by 14

50 most cited of 140 All 140 (JSON)