mlsanityData to 7 Oct 2026

Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities

arXiv 2505.17862 ↗First posted 23 May 2025cs.AI cs.CL cs.CVcited by 101

On the map: LLM reasoning → Audio-language model reasoning

also in 1 more
Abstract (arXiv)

Recent Multimodal Large Language Models (MLLMs) achieve promising performance on visual and audio benchmarks independently. However, the ability of these models to process cross-modal information synchronously remains largely unexplored. We introduce Daily-Omni, a multiple-choice Audio-Visual QA benchmark featuring 684 real-world videos and 1,197 questions spanning 6 task families that explicitly require cross-modal temporal reasoning. To support scalable benchmark construction, we develop a semi-automatic pipeline for annotation, cross-modal consistency refinement, temporal alignment elicitation, and text-only leakage filtering, followed by human verification. We further provide a diagnostic evaluation suite and extensively evaluate 24 foundation models under 37 model--modality settings (Audio+Video / Audio-only / Video-only / Text-only). Finally, we include a training-free modular diagnostic baseline that composes off-the-shelf unimodal models to serve as a diagnostic baseline and to illustrate how explicit temporal alignment signals affect performance. Results indicate that many end-to-end MLLMs still struggle on alignment-critical questions, suggesting that robust cross-modal temporal alignment remains an important open challenge.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 55 of its 70 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

55 of 70 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Qwen2.5-Omni Technical Report26 Mar
  2. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities6 Mar
  3. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities17 Jun

Other references · 52

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  12. 2024Emu3: Next-Token Prediction is All You Needcited by 761
38 more
  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  3. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  4. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  5. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  6. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  7. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  8. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  9. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  10. 2024Qwen2-Audio Technical Reportcited by 543
  11. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  12. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  13. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  14. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  15. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  16. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  17. 2025Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videoscited by 225
  18. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 215
  19. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  20. 2023Listen, Think, and Understandcited by 206
  21. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  22. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  23. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  24. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  25. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  26. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  27. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  28. 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilitiescited by 104
  29. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
  30. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 100
  31. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  32. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  33. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Modelscited by 67
  34. 2022Vocalsound: A Dataset for Improving Human Vocal Sounds Recognitioncited by 62
  35. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 50
  36. 2024LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videoscited by 49
  37. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  38. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?cited by 46

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 20; 2026: 81.

101 papers.

Sort:

By area:

  1. 2025Qwen3-Omni Technical Reportcited by 451
  2. 2026Qwen3.5-Omni Technical Reportcited by 146
  3. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 64
  4. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 54
  5. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 53
  6. 2025HumanOmniV2: From Understanding to Omni-Modal Reasoning with Contextcited by 50
  7. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 39
  8. 2025AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestrationcited by 26
  9. 2025Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perceptioncited by 25
  10. 2026Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligencecited by 25
  11. 2025LongCat-Flash-Omni Technical Reportcited by 23
  12. 2026OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Modelscited by 23
38 more
  1. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  2. 2026OmniGAIA: Towards Native Omni-Modal AI Agentscited by 21
  3. 2025See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Modelscited by 19
  4. 2026MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videoscited by 14
  5. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
  6. 2025UNO-Bench: A Unified Benchmark for Exploring the Compositional Law Between Uni-modal and Omni-modal in Omni Modelscited by 12
  7. 2026SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Modelscited by 12
  8. 2025AVATAR: Reinforcement Learning to See, Hear, and Reason Over Videocited by 9
  9. 2026OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attentioncited by 9
  10. 2026Do Audio-Visual Large Language Models Really See and Hear?cited by 9
  11. 2026TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captionscited by 8
  12. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
  13. 2026FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMscited by 7
  14. 2026Towards Universal Video MLLMs with Attribute-Structured and Quality-Verified Instructionscited by 7
  15. 2026D-ORCA: Dialogue-Centric Optimization for Robust Audio-Visual Captioningcited by 6
  16. 2026Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Groundingcited by 6
  17. 2025FysicsWorld: A Unified Full-Modality Benchmark for Any-to-Any Understanding, Generation, and Reasoningcited by 5
  18. 2025A Benchmark for Omni-Modal Reasoning in Long Videoscited by 5
  19. 2026OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answeringcited by 5
  20. 2026OmniFysics: Towards Physical Intelligence Evolution via Omni-Modal Signal Processing and Network Optimizationcited by 5
  21. 2026On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMscited by 5
  22. 2026LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoningcited by 5
  23. 2026OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chainscited by 5
  24. 2025XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Modelscited by 4
  25. 2025EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMscited by 4
  26. 2026ThinkOmni: Lifting Textual Reasoning to Omni-modal Scenarios via Guidance Decodingcited by 4
  27. 2026StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggeringcited by 4
  28. 2025LongInsightBench: A Comprehensive Benchmark for Evaluating Omni-Modal Models on Human-Centric Long-Video Understandingcited by 3
  29. 2026SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understandingcited by 3
  30. 2026MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimizationcited by 3
  31. 2026Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMscited by 3
  32. 2026Native Active Perception as Reasoning for Omni-Modal Understandingcited by 3
  33. 2026OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Modelscited by 3
  34. 2026Multimodal Duplex Interaction Agentcited by 3
  35. 2026When Vision Speaks for Soundcited by 2
  36. 2026Agentic Active Omni-Modal Perception for Multi-Hop Audio-Visual Reasoningcited by 2
  37. 2026AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compressioncited by 2
  38. 2026OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Modelscited by 2

50 most cited of 101 All 101 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation
  2. 2026OmniReasoning: Pushing the Limits of Audio-Visual Joint Reasoning
  3. 2026Temporal and Cross-Modal Alignment for Enhanced Audiovisual Video Captioning
  4. 2026Omni-Judge: Can Omni-LLMs Serve as Human-Aligned Judges for Text-Conditioned Audio-Video Generation?
  5. 2025Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
  6. 2026C3PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models
  7. 2024X-VILA: Cross-Modality Alignment for Large Language Model
  8. 2025AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understanding