mlsanityData to 7 Oct 2026

WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs

arXiv 2502.04326 ↗First posted 6 Feb 2025cs.CV cs.AIcited by 147

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

We introduce WorldSense, the first benchmark to assess the multi-modal video understanding, that simultaneously encompasses visual, audio, and text inputs. In contrast to existing benchmarks, our WorldSense has several features: (i)collaboration of omni-modality, we design the evaluation tasks to feature a strong coupling of audio and video, requiring models to effectively utilize the synergistic perception of omni-modality; (ii)diversity of videos and tasks, WorldSense encompasses a diverse collection of 1,662 audio-visual synchronised videos, systematically categorized into 8 primary domains and 67 fine-grained subcategories to cover the broad scenarios, and 3,172 multi-choice QA pairs across 26 distinct tasks to enable the comprehensive evaluation; (iii)high-quality annotations, all the QA pairs are manually labeled by 80 expert annotators with multiple rounds of correction to ensure quality. Based on our WorldSense, we extensively evaluate various state-of-the-art models. The experimental results indicate that existing models face significant challenges in understanding real-world scenarios (65.1% best accuracy). By analyzing the limitations of current models, we aim to provide valuable insight to guide development of real-world understanding. We hope our WorldSense can provide a platform for evaluating the ability in constructing and understanding coherent contexts from omni-modality.

Key facts. Cited by 147 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 89 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

55 of 89 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?3 Dec
  2. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding6 Nov
  3. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture4 Sep
  4. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models9 Aug
  5. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output3 Jul
  6. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding20 Jun
  7. 2024LVBench: An Extreme Long Video Understanding Benchmark12 Jun
  8. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  9. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov

Other references · 46

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  11. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  12. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
34 more
  1. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  3. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  4. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  5. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  6. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  7. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  8. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  9. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  10. 2024Qwen2-Audio Technical Reportcited by 543
  11. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  12. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  13. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  14. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  15. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  16. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  17. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 123
  18. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  19. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  20. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  21. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
  22. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 100
  23. 2024MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answeringcited by 93
  24. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89
  25. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  26. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Modelscited by 67
  27. 2024LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videoscited by 49
  28. 2024MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videoscited by 43
  29. 2024VILA2: VILA Augmented VILAcited by 25
  30. 2024Visual Perception by Large Language Model's Weightscited by 19
  31. 2024VCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Textcited by 15
  32. 2024Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilitiescited by 15
  33. 2023InstructSeq: Unifying Vision Tasks with Instruction-conditioned Multi-modal Sequence Generationcited by 14
  34. 2024EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Modelcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 46; 2026: 101.

147 papers.

Sort:

By area:

  1. 2025Qwen3-Omni Technical Reportcited by 451
  2. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  3. 2026Qwen3.5-Omni Technical Reportcited by 146
  4. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
  5. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  6. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  7. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 64
  8. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 55
  9. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 54
  10. 2025HumanOmniV2: From Understanding to Omni-Modal Reasoning with Contextcited by 50
  11. 2025UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encodingcited by 49
  12. 2025MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoningcited by 45
38 more
  1. 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidencecited by 43
  2. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 39
  3. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  4. 2025AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestrationcited by 26
  5. 2025Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perceptioncited by 25
  6. 2026Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligencecited by 25
  7. 2025Sketch-in-Latents: Eliciting Unified Reasoning in MLLMscited by 24
  8. 2025LongCat-Flash-Omni Technical Reportcited by 23
  9. 2026OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Modelscited by 23
  10. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  11. 2025Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Maskingcited by 21
  12. 2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 21
  13. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  14. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 21
  15. 2026OmniGAIA: Towards Native Omni-Modal AI Agentscited by 21
  16. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  17. 2025See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Modelscited by 19
  18. 2025AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMscited by 15
  19. 2026MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videoscited by 14
  20. 2025OmniEval: A Benchmark for Evaluating Omni-modal Models with Visual, Auditory, and Textual Inputscited by 13
  21. 2025JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluationcited by 13
  22. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
  23. 2025Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Visioncited by 12
  24. 2025UNO-Bench: A Unified Benchmark for Exploring the Compositional Law Between Uni-modal and Omni-modal in Omni Modelscited by 12
  25. 2026SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Modelscited by 12
  26. 2025CrossLMM: Decoupling Long Video Sequences from LMMs via Dual Cross-Attention Mechanismscited by 11
  27. 2025AVATAR: Reinforcement Learning to See, Hear, and Reason Over Videocited by 9
  28. 2026Do Audio-Visual Large Language Models Really See and Hear?cited by 9
  29. 2026VISD: Enhancing Video Reasoning via Structured Self-Distillationcited by 9
  30. 2025CyberV: Cybernetics for Test-time Scaling in Video Understandingcited by 8
  31. 2025MAVERIX: Multimodal Audio-Visual Evaluation and Recognition IndeXcited by 7
  32. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
  33. 2026FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMscited by 7
  34. 2026MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Mergingcited by 7
  35. 2026Towards Universal Video MLLMs with Attribute-Structured and Quality-Verified Instructionscited by 7
  36. 2026D-ORCA: Dialogue-Centric Optimization for Robust Audio-Visual Captioningcited by 6
  37. 2026DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compressioncited by 6
  38. 2026Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstentioncited by 6

50 most cited of 147 All 147 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024WorldQA: Multimodal World Knowledge in Videos through Long-Chain Reasoning
  2. 2026Can MiniMax-H3 Reason About the Physical World? An Evaluation of Omni-Modal Generative Model
  3. 2025Audio-centric Video Understanding Benchmark without Text Shortcut
  4. 2026Toward Native Multimodal Modeling: A Roadmap
  5. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
  6. 2026MVPBench: A Multi-Video Perception Evaluation Benchmark for Multi-Modal Video Understanding
  7. 2023Audio-Visual LLM for Video Understanding
  8. 2025VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs