# Robust Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning

arXiv: https://arxiv.org/abs/2502.12425 · HTML: https://mlsanity.com/paper/2502.12425/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning.
- Builds on 44 of its 73 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Placed in

- LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/)

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [A Corpus for Reasoning About Natural Language Grounded in Photographs](https://mlsanity.com/paper/1811.00491/) (2018)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [Pengi: An Audio Language Model for Audio Tasks](https://mlsanity.com/paper/2305.11834/) (2023)
- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound](https://mlsanity.com/paper/2201.02639/) (2022)
- [ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2304.03047/) (2023)
- [Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling](https://mlsanity.com/paper/2307.14126/) (2023)
- [Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification](https://mlsanity.com/paper/2108.08728/) (2021)
- [Visual Commonsense R-CNN](https://mlsanity.com/paper/2002.12204/) (2020)
- [Are Disentangled Representations Helpful for Abstract Visual Reasoning?](https://mlsanity.com/paper/1905.12506/) (2019)
- [Disentangled Person Image Generation](https://mlsanity.com/paper/1712.02621/) (2017)
- [Semantic Audio-Visual Navigation](https://mlsanity.com/paper/2012.11583/) (2020)
- [Attentive Relational Networks for Mapping Images to Scene Graphs](https://mlsanity.com/paper/1811.10696/) (2018)
- [S3VAE: Self-Supervised Sequential VAE for Representation Disentanglement and Data Generation](https://mlsanity.com/paper/2005.11437/) (2020)
- [Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering](https://mlsanity.com/paper/2207.12647/) (2022)

## Built on by (most cited first, 20 of 20)

- [DC-SAM: In-Context Segment Anything in Images and Videos via Dual Consistency](https://mlsanity.com/paper/2504.12080/) (2025)
- [Explainable Action Form Assessment by Exploiting Multimodal Chain-of-Thoughts Reasoning](https://mlsanity.com/paper/2512.15153/) (2025)
- [RoboDriveVLM: A Novel Benchmark and Baseline towards Robust Vision-Language Models for Autonomous Driving](https://mlsanity.com/paper/2512.01300/) (2025)
- [Robo-SGG: Exploiting Layout-Oriented Normalization and Restitution Can Improve Robust Scene Graph Generation](https://mlsanity.com/paper/2504.12606/) (2025)
- [Active Exploring like a Pigeon: Reinforcing Spatial Reasoning via Agentic Vision-Language Models](https://mlsanity.com/paper/2606.02459/) (2026)
- [Closing the Reality Gap: Zero-Shot Sim-to-Real Deployment for Dexterous Force-Based Grasping and Manipulation](https://mlsanity.com/paper/2601.02778/) (2026)
- [CritiqueDriveVLM: From Verifier-Guided Reinforcement Learning to Latent Thought Distillation for Autonomous Driving](https://mlsanity.com/paper/2607.04179/) (2026)
- [Chain-of-Evidence Multimodal Reasoning for Few-shot Temporal Action Localization](https://mlsanity.com/paper/2504.13460/) (2025)
- [Reason-Then-Retrieve for CoVR-R with Structured Edit Prompts and Dense-Sparse Fusion](https://mlsanity.com/paper/2606.02450/) (2026)
- [A VideoMAE-v2 Approach to Zero-Shot Traffic Accident Anticipation](https://mlsanity.com/paper/2606.09542/) (2026)
- [Uncovering and Mitigating Transient Blindness in Multimodal Model Editing](https://mlsanity.com/paper/2511.13243/) (2025)
- [Multi-Modal Scene Graph with Kolmogorov-Arnold Experts for Audio-Visual Question Answering](https://mlsanity.com/paper/2511.23304/) (2025)
- [Question-Aware Evidence Ledgers for Video Relational Reasoning](https://mlsanity.com/paper/2606.02506/) (2026)
- [Claude Code-Driving Scenario Mining for the Argoverse 2 Challenge](https://mlsanity.com/paper/2606.09180/) (2026)
- [Leveraging Metric Depth for Relative Depth Prediction](https://mlsanity.com/paper/2606.10628/) (2026)
- [SGFormer++: Semantic Graph Transformer for Incremental 3D Scene Graph Generation](https://mlsanity.com/paper/2606.15328/) (2026)
- [A DVDrive Approach for doScenes Instructed Driving Challenge](https://mlsanity.com/paper/2606.21623/) (2026)
- [Boosting Robustness for All-Weather Self-Supervised Depth Estimation in Autonomous Driving](https://mlsanity.com/paper/2607.21526/) (2026)
- [VTO: Visual Tool Orchestration for Video Anomaly Detection](https://mlsanity.com/paper/2608.08219/) (2026)
- [ExPhy: A Benchmark for Explicit Physical Property Learning in Multi-Object Trajectory Forecasting](https://mlsanity.com/paper/2608.20009/) (2026)

## Abstract (arXiv)

In this paper, we propose a new Robust Disentangled Counterfactual Learning (RDCL) approach for physical audiovisual commonsense reasoning. The task aims to infer objects' physics commonsense based on both video and audio input, with the main challenge being how to imitate the reasoning ability of humans, even under the scenario of missing modalities. Most of the current methods fail to take full advantage of different characteristics in multi-modal data, and lacking causal reasoning ability in models impedes the progress of implicit physical knowledge inferring. To address these issues, our proposed RDCL method decouples videos into static (time-invariant) and dynamic (time-varying) factors in the latent space by the disentangled sequential encoder, which adopts a variational autoencoder (VAE) to maximize the mutual information with a contrastive loss function. Furthermore, we introduce a counterfactual learning module to augment the model's reasoning ability by modeling physical knowledge relationships among different objects under counterfactual intervention. To alleviate the incomplete modality data issue, we introduce a robust multimodal learning method to recover the missing data by decomposing the shared features and model-specific features. Our proposed method is a plug-and-play module that can be incorporated into any baseline including VLMs. In experiments, we show that our proposed method improves the reasoning accuracy and robustness of baseline methods and achieves the state-of-the-art performance.