Do Joint Audio-Video Generation Models Understand Physics?
Not on the map. Why?
Abstract (arXiv)
Joint audio-video generation models are rapidly approaching professional production quality, raising a central question: do they understand audio-visual physics, or merely generate plausible sounds and frames that violate real-world consistency? We introduce AV-Phys Bench, a benchmark for evaluating physical commonsense in joint audio-video generation. AV-Phys Bench tests models across three scene categories: Steady State, Event Transition, and Environment Transition. It covers physics-grounded subcategories drawn from real-world scenes, plus Anti-AV-Physics prompts that deliberately request physically inconsistent audio-video behavior. Each generation is evaluated along five dimensions: visual semantic adherence, audio semantic adherence, visual physical commonsense, audio physical commonsense, and cross-modal physical commonsense. Across three proprietary and four open-source models, we find that Seedance 2.0 performs best overall, but all models remain far from robust physical understanding. Performance drops sharply on event-driven and environment-driven transitions, and even strong proprietary systems collapse on Anti-AV-Physics prompts. We further introduce AV-Phys Agent, a ReAct-style evaluator that combines a multimodal language model with deterministic acoustic measurement tools, producing rankings that closely align with human ratings. Our results identify cross-modal physical consistency and transition-driven scene dynamics as key open challenges for joint audio-video generation.
Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 41 of its 47 references that are in the map. First posted on arXiv 8 May 2026.
Builds on
41 of 47 references.
References · 41
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
- 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
- 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
- 2025OpenAI GPT-5 System Cardcited by 989
- 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
- 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
- 2024Genie: Generative Interactive Environmentscited by 747
- 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 236
- 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 228
- 2023A Survey on Video Diffusion Modelscited by 206
29 more
- 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 197
- 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 185
- 2024Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generationcited by 174
- 2026Qwen3.5-Omni Technical Reportcited by 174
- 2025Do generative video models understand physical principles?cited by 161
- 2026Seedance 2.0: Advancing Video Generation for World Complexitycited by 160
- 2025VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generationcited by 149
- 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 96
- 2022SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learningcited by 95
- 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 88
- 2025Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generationcited by 83
- 2023Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptationcited by 69
- 2025UniVerse-1: Unified Audio-Video Generation via Stitching of Expertscited by 61
- 2025T2VPhysBench: A First-Principles Benchmark for Physical Consistency in Text-to-Video Generationcited by 39
- 2024TAVGBench: Benchmarking Text to Audible-Video Generationcited by 34
- 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
- 2025VABench: A Comprehensive Benchmark for Audio-Video Generationcited by 30
- 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 25
- 2026Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Modelcited by 25
- 2025T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generationcited by 24
- 2024From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generationcited by 23
- 2025SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearingcited by 15
- 2023CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environmentscited by 14
- 2025PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generationcited by 11
- 2024SAVGBench: Benchmarking Spatially Aligned Audio-Video Generationcited by 10
- 2026A Systematic Post-Train Framework for Video Generationcited by 5
- 2024Virbo: Multimodal Multilingual Avatar Video Generation in Digital Marketingcited by 4
- 2026SonicBench: Dissecting the Physical Perception Bottleneck in Large Audio Language Modelscited by 2
- 2026OmniSonic: Towards Universal and Holistic Audio Generation from Video and Textcited by 1
Built on by
Citing papers per year: 2026: 8.
8 papers.
- 2026The Attention Triangle in Audio-Video Modelscited by 1
- 2026Beyond Coherence: Benchmarking Professional Editing-Technique Execution in Multi-Shot Audio-Video Generationcited by 1
- 2026VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generationcited by 1
- 2026VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinementcited by 0
- 2026PIVOT: Physics-Grounded Verification for AI-Generated Audio-Video Detectioncited by 0
- 2026Can MiniMax-H3 Reason About the Physical World? An Evaluation of Omni-Modal Generative Modelcited by 0
- 2026Intervention, Not Shared Latents: Blocking Visual Shortcuts in Audio-Video Generationcited by 0
- 2026Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomycited by 0
- 2026VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation1 Oct 2026
- 2026Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomy28 Sep 2026
- 2026Can MiniMax-H3 Reason About the Physical World? An Evaluation of Omni-Modal Generative Model16 Sep 2026
- 2026PIVOT: Physics-Grounded Verification for AI-Generated Audio-Video Detection14 Sep 2026
- 2026Beyond Coherence: Benchmarking Professional Editing-Technique Execution in Multi-Shot Audio-Video Generation8 Sep 2026
- 2026The Attention Triangle in Audio-Video Models3 Sep 2026
- 2026VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement2 Sep 2026
- 2026Intervention, Not Shared Latents: Blocking Visual Shortcuts in Audio-Video Generation14 Aug 2026
Similar, not cited
Close in wording, no citation either way
- 2025"PhyWorldBench": A Comprehensive Evaluation of Physical Realism in Text-to-Video Models
- 2025PAVAS: Physics-Aware Video-to-Audio Synthesis
- 2025PhysVidBench: Language-Grounded Evaluation of Physical Commonsense in Text-to-Video Models
- 2025Seedance 1.5 pro: A Native Audio-Visual Joint Generation Foundation Model
- 2026AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation
- 2025Bootstrapping Physics-Grounded Video Generation through VLM-Guided Iterative Self-Refinement
- 2026AVBench: Human-Aligned and Automated Evaluation Benchmark for Audio-Video Generative Models
- 2026Physically Plausible Video Generation via Visual-Semantic Chain-of-Events Conditioning