mlsanityData to 14 Oct 2026

Pegasus-v1 Technical Report

arXiv 2404.14687 ↗First posted 23 Apr 2024cs.MM cs.AI cs.CLcited by 4

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

This technical report introduces Pegasus-1, a multimodal language model specialized in video content understanding and interaction through natural language. Pegasus-1 is designed to address the unique challenges posed by video data, such as interpreting spatiotemporal information, to offer nuanced video content comprehension across various lengths. This technical report overviews Pegasus-1's architecture, training strategies, and its performance in benchmarks on video conversation, zero-shot video question answering, and video summarization. We also explore qualitative characteristics of Pegasus-1 , demonstrating its capabilities as well as its limitations, in order to provide readers a balanced view of its current state and its future direction.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 21 references that are in the map. First posted on arXiv 23 Apr 2024.

Builds on

16 of 21 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 13

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2023Mistral 7Bcited by 3,164
  6. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  7. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  8. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  9. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  10. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  11. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  12. 2022Linearly Mapping from Image to Text Spacecited by 142
1 more
  1. 2023BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuningcited by 41

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 1.

4 papers.

  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  2. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  3. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  4. 2024HourVideo: 1-Hour Video-Language Understandingcited by 113

Similar, not cited

Close in wording, no citation either way

  1. 2025An Empirical Study for Representations of Videos in Video Question Answering via MLLMs
  2. 2024Artemis: Towards Referential Understanding in Complex Videos
  3. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding
  4. 2023VaQuitA: Enhancing Alignment in LLM-Assisted Video Understanding
  5. 2025SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models
  6. 2026LEMON: How Well Do MLLMs Perform Temporal Multimodal Understanding on Instructional Videos?
  7. 2025SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding
  8. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens