mlsanityData to 7 Oct 2026

LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos

arXiv 2411.19772 ↗First posted 29 Nov 2024cs.CV cs.CL cs.LGcited by 49

Not on the map. Why?

Abstract (arXiv)

Despite impressive advancements in video understanding, most efforts remain limited to coarse-grained or visual-only video tasks. However, real-world videos encompass omni-modal information (vision, audio, and speech) with a series of events forming a cohesive storyline. The lack of multi-modal video data with fine-grained event annotations and the high cost of manual labeling are major obstacles to comprehensive omni-modality video perception. To address this gap, we propose an automatic pipeline consisting of high-quality multi-modal video filtering, semantically coherent omni-modal event boundary detection, and cross-modal correlation-aware event captioning. In this way, we present LongVALE, the first-ever Vision-Audio-Language Event understanding benchmark comprising 105K omni-modal events with precise temporal boundaries and detailed relation-aware captions within 8.4K high-quality long videos. Further, we build a baseline that leverages LongVALE to enable video large language models (LLMs) for omni-modality fine-grained temporal video understanding for the first time. Extensive experiments demonstrate the effectiveness and great potential of LongVALE in advancing comprehensive multi-modal video understanding.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 46 of its 63 references that are in the map. First posted on arXiv 29 Nov 2024.

Builds on

46 of 63 references.

References · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  10. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  11. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  12. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
34 more
  1. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  2. 2017Dense-Captioning Events in Videoscited by 789
  3. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  4. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  5. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  6. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  7. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  8. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  9. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  10. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  11. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  12. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  13. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  14. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  15. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  16. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  17. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  18. 2023UniVTG: Towards Unified Video-Language Temporal Groundingcited by 189
  19. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  20. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  21. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  22. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  23. 2022UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detectioncited by 148
  24. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  25. 2021End-to-End Dense Video Captioning with Parallel Decodingcited by 133
  26. 2019Audio-Visual Scene-Aware Dialogcited by 122
  27. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  28. 2023Audio-Visual LLM for Video Understandingcited by 76
  29. 2021ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learningcited by 50
  30. 2023Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baselinecited by 45
  31. 2023Connecting Multi-modal Contrastive Representationscited by 36
  32. 2023Towards Generalisable Video Moment Retrieval: Visual-Dynamic Injection to Image-Text Pre-Trainingcited by 32
  33. 2018Speech waveform synthesis from MFCC sequences with generative adversarial networkscited by 5
  34. 2024VidLA: Video-Language Alignment at Scalecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 20; 2026: 28.

49 papers.

Sort:

By area:

  1. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  2. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  3. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  4. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  5. 2025Audio-centric Video Understanding Benchmark without Text Shortcutcited by 33
  6. 2025AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestrationcited by 26
  7. 2025Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perceptioncited by 25
  8. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  9. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  10. 2025Vidi: Large Multimodal Models for Video Understanding and Editingcited by 14
  11. 2025SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearingcited by 14
  12. 2025CVBench: Benchmarking Cross-Video Synergies for Complex Multimodal Reasoningcited by 14
37 more
  1. 2025OmniEval: A Benchmark for Evaluating Omni-modal Models with Visual, Auditory, and Textual Inputscited by 13
  2. 2025JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluationcited by 13
  3. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 11
  4. 2025AdaRD-key: Adaptive Relevance-Diversity Keyframe Sampling for Long-form Video understandingcited by 9
  5. 2026LiViBench: An Omnimodal Benchmark for Interactive Livestream Video Understandingcited by 8
  6. 2026TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captionscited by 8
  7. 2025Watch and Listen: Understanding Audio-Visual-Speech Moments with Multimodal LLMcited by 7
  8. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
  9. 2026FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMscited by 7
  10. 2025ChronusOmni: Improving Time Awareness of Omni Large Language Modelscited by 5
  11. 2025EchoFoley: Event-Centric Hierarchical Control for Video Grounded Creative Sound Generationcited by 5
  12. 2026OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Videocited by 5
  13. 2025PreFM: Online Audio-Visual Event Parsing via Predictive Future Modelingcited by 4
  14. 2026SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understandingcited by 3
  15. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 3
  16. 2026Native Active Perception as Reasoning for Omni-Modal Understandingcited by 3
  17. 2026MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Modelscited by 2
  18. 2026EgoAVU: Egocentric Audio-Visual Understandingcited by 2
  19. 2026OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Modelscited by 2
  20. 2026Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videoscited by 2
  21. 2025Multimodal Long Video Modeling Based on Temporal Dynamic Contextcited by 1
  22. 2026Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Modelscited by 1
  23. 2026Valley3: Scaling Omni Foundation Models for E-commercecited by 1
  24. 2026VideoOdyssey: A Benchmark for Ultra-Long-Context and Omni-Modal Video Understandingcited by 1
  25. 2026CapRiCorn-1K: A Comprehensive Benchmark for Video Captioning and Subject Referential Consistency Across Temporal Scalescited by 1
  26. 2026AVTrace: Diagnosing Audio-Visual Temporal Reasoning in Omni Modelscited by 1
  27. 2026EAR: Enhancing Uni-Modal Representations for Weakly Supervised Audio-Visual Video Parsingcited by 0
  28. 2026Parallelized Autoregressive Decoding for Omni-Modal Dense Video Captioningcited by 0
  29. 2026Empowering Long-form Omni-modal Understanding with Robust Audio Perceptioncited by 0
  30. 2026TimePLE: Rethinking Temporal Representation for Video Temporal Groundingcited by 0
  31. 2026AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Rewardcited by 0
  32. 2026Can Language Models Understand mmWave Data? Benchmarking Large Language Models for mmWave Radar-Based Human Understandingcited by 0
  33. 2026Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarioscited by 0
  34. 2026OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialoguecited by 0
  35. 2026Agentic Video Understanding: A Surveycited by 0
  36. 2026OmniTide: Co-Designing Algorithms and Systems for Efficient On-Device Omni-LLM Streamingcited by 0
  37. 2026LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perceptioncited by 0