mlsanityData to 7 Oct 2026

E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding

arXiv 2409.18111 ↗First posted 26 Sep 2024cs.CVcited by 68

Not on the map. Why?

Cited by 23 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in Video Large Language Models (Video-LLMs) have demonstrated their great potential in general-purpose video understanding. To verify the significance of these models, a number of benchmarks have been proposed to diagnose their capabilities in different scenarios. However, existing benchmarks merely evaluate models through video-level question-answering, lacking fine-grained event-level assessment and task diversity. To fill this gap, we introduce E.T. Bench (Event-Level & Time-Sensitive Video Understanding Benchmark), a large-scale and high-quality benchmark for open-ended event-level video understanding. Categorized within a 3-level task taxonomy, E.T. Bench encompasses 7.3K samples under 12 tasks with 7K videos (251.4h total length) under 8 domains, providing comprehensive evaluations. We extensively evaluated 8 Image-LLMs and 12 Video-LLMs on our benchmark, and the results reveal that state-of-the-art models for coarse-level (video-level) understanding struggle to solve our fine-grained tasks, e.g., grounding event-of-interests within videos, largely due to the short video context length, improper time representations, and lack of multi-event training data. Focusing on these issues, we further propose a strong baseline model, E.T. Chat, together with an instruction-tuning dataset E.T. Instruct 164K tailored for fine-grained event-level understanding. Our simple but effective solution demonstrates superior performance in multiple scenarios.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 88 of its 111 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

88 of 111 references.

References · 88

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2023Segment Anythingcited by 8,158
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
38 more
  1. 2023Qwen Technical Reportcited by 3,273
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  12. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  13. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  14. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  15. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  16. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  17. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  18. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  19. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  20. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  21. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  22. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  23. 2017Dense-Captioning Events in Videoscited by 789
  24. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  25. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  26. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  27. 2017Localizing Moments in Video with Natural Languagecited by 660
  28. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  29. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  30. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  31. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  32. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  33. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  34. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  35. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  36. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  37. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  38. 2024InternLM2 Technical Reportcited by 391

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 43; 2026: 19.

68 papers.

Sort:

By area:

  1. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 134
  2. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
  3. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 90
  4. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 87
  5. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 82
  6. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoningcited by 66
  7. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  8. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
  9. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
  10. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 38
  11. 2025Generative Frame Sampler for Long Video Understandingcited by 35
  12. 2025DisasterM3: A Remote Sensing Vision-Language Dataset for Disaster Damage Assessment and Responsecited by 35
38 more
  1. 2025MINERVA: Evaluating Complex Video Reasoningcited by 31
  2. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 28
  3. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  4. 2025Streaming Video Instruction Tuningcited by 25
  5. 2025VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videoscited by 24
  6. 2025ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Modelscited by 24
  7. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 21
  8. 2024LinVT: Empower Your Image-level Large Language Model to Understand Videoscited by 19
  9. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Modelcited by 19
  10. 2025DynamicVL: Benchmarking Multimodal Large Language Models for Dynamic City Understandingcited by 17
  11. 2025RSCC: A Large-Scale Remote Sensing Change Caption Dataset for Disaster Eventscited by 13
  12. 2025MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Groundingcited by 11
  13. 2025ELV-Halluc: Benchmarking Semantic Aggregation Hallucinations in Long Video Understandingcited by 10
  14. 2024Exploring What Why and How: A Multifaceted Benchmark for Causation Understanding of Video Anomalycited by 9
  15. 2025TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videoscited by 9
  16. 2025AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understandingcited by 9
  17. 2024INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuningcited by 8
  18. 2025GLIMPSE: Do Large Vision-Language Models Truly Think With Videos or Just Glimpse at Them?cited by 8
  19. 2025DisTime: Distribution-based Time Representation for Video Large Language Modelscited by 7
  20. 2025VideoTG-R1: Boosting Video Temporal Grounding via Curriculum Reinforcement Learning on Reflected Boundary Annotationscited by 7
  21. 2025TempR1: Improving Temporal Understanding of MLLMs via Temporal-Aware Multi-Task Reinforcement Learningcited by 7
  22. 2025Open-ended Hierarchical Streaming Video Understanding with Vision Language Modelscited by 6
  23. 2025SeriesBench: A Benchmark for Narrative-Driven Drama Series Understandingcited by 5
  24. 2025SVLTA: Benchmarking Vision-Language Temporal Alignment via Synthetic Video Situationcited by 4
  25. 2025MUPA: Towards Multi-Path Agentic Reasoning for Grounded Video Question Answeringcited by 4
  26. 2025MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understandingcited by 4
  27. 2025ExpVid: A Benchmark for Experiment Video Understanding & Reasoningcited by 4
  28. 2025TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understandingcited by 4
  29. 2025Factorized Learning for Temporally Grounded Video-Language Modelscited by 4
  30. 2025Looking Beyond Visible Cues: Implicit Video Question Answering via Dual-Clue Reasoningcited by 3
  31. 2025UVLM: Benchmarking Video Language Model for Underwater World Understandingcited by 3
  32. 2025NeMo: Needle in a Montage for Video-Language Understandingcited by 3
  33. 2025Video-in-the-Loop: Span-Grounded Long Video QA with Interleaved Reasoningcited by 3
  34. 2026EvoGround: Self-Evolving Video Agents for Video Temporal Groundingcited by 3
  35. 2026StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Videocited by 3
  36. 2026An Efficient Streaming Video Understanding Framework with Agentic Controlcited by 3
  37. 2026Kwai Keye-VL-2.0 Technical Reportcited by 3
  38. 2025Measure Twice, Cut Once: A Semantic-Oriented Approach to Video Temporal Localization with Video LLMscited by 2

50 most cited of 68 All 68 (JSON)