# CAUSAL3D: A Comprehensive Benchmark for Causal Learning from Visual Data

arXiv: https://arxiv.org/abs/2503.04852 · HTML: https://mlsanity.com/paper/2503.04852/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 53 references that are in the map.
- First posted on arXiv 6 Mar 2025.

## Builds on (most cited first, 30 of 39)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [DAGs with NO TEARS: Continuous Optimization for Structure Learning](https://mlsanity.com/paper/1803.01422/) (2018)
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](https://mlsanity.com/paper/2211.05778/) (2022)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [DAG-GNN: DAG Structure Learning with Graph Neural Networks](https://mlsanity.com/paper/1904.10098/) (2019)
- [Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style](https://mlsanity.com/paper/2106.04619/) (2021)
- [Detecting causal associations in large nonlinear time series datasets](https://mlsanity.com/paper/1702.07007/) (2017)
- [Causality for Machine Learning](https://mlsanity.com/paper/1911.10500/) (2019)
- [Gradient-Based Neural DAG Learning](https://mlsanity.com/paper/1906.02226/) (2019)
- [Contrastive Learning Inverts the Data Generating Process](https://mlsanity.com/paper/2102.08850/) (2021)
- [CAM: Causal additive models, high-dimensional order search and penalized regression](https://mlsanity.com/paper/1310.1533/) (2013)
- [Revisiting Weakly Supervised Pre-Training of Visual Perception Models](https://mlsanity.com/paper/2201.08371/) (2022)
- [Causal Attention for Vision-Language Tasks](https://mlsanity.com/paper/2103.03493/) (2021)
- [Score matching enables causal discovery of nonlinear additive noise models](https://mlsanity.com/paper/2203.04413/) (2022)
- [Learning Equivalence Classes of Bayesian Networks Structures](https://mlsanity.com/paper/1302.3566/) (2013)
- [Weakly Supervised Disentangled Generative Causal Representation Learning](https://mlsanity.com/paper/2010.02637/) (2020)
- [Diffusion Models for Causal Discovery via Topological Ordering](https://mlsanity.com/paper/2210.06201/) (2022)
- [Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties](https://mlsanity.com/paper/2306.15668/) (2023)
- [Efficient Causal Graph Discovery Using Large Language Models](https://mlsanity.com/paper/2402.01207/) (2024)
- [Vision-and-Language Navigation via Causal Learning](https://mlsanity.com/paper/2404.10241/) (2024)
- [CausalVAE: Structured Causal Disentanglement in Variational Autoencoder](https://mlsanity.com/paper/2004.08697/) (2020)
- [CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractions](https://mlsanity.com/paper/2012.04293/) (2020)
- [Causal Inference with Large Language Model: A Survey](https://mlsanity.com/paper/2409.09822/) (2024)
- [Causality for Large Language Models](https://mlsanity.com/paper/2410.15319/) (2024)
- [Causal Reasoning Meets Visual Representation Learning: A Prospective Study](https://mlsanity.com/paper/2204.12037/) (2022)
- [Learning Causally Disentangled Representations via the Principle of Independent Causal Mechanisms](https://mlsanity.com/paper/2306.01213/) (2023)
- [Improving Certified Robustness via Statistical Learning with Logical Reasoning](https://mlsanity.com/paper/2003.00120/) (2020)
- [Causal-TGAN: Generating Tabular Data Using Causal Generative Adversarial Networks](https://mlsanity.com/paper/2104.10680/) (2021)

## Built on by (most cited first, 7 of 7)

- [CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models](https://mlsanity.com/paper/2506.11034/) (2025)
- [CausalVerse: Benchmarking Causal Representation Learning with Configurable High-Fidelity Simulations](https://mlsanity.com/paper/2510.14049/) (2025)
- [Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI](https://mlsanity.com/paper/2510.04978/) (2025)
- [Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs](https://mlsanity.com/paper/2606.05966/) (2026)
- [Distilling Counterfactual Reasoning from Language to Vision: Causal Graph Guided Post-Training for Video Understanding](https://mlsanity.com/paper/2511.19923/) (2025)
- [From Prompts to Tokens: Internalizing Causal Supervision in Vision-Language Model for Multi-Image Causal Reasoning](https://mlsanity.com/paper/2606.11745/) (2026)
- [CCRV-Bench: Constraint-Based Evaluation of Causal Reasoning in Vision-Language Models](https://mlsanity.com/paper/2609.30979/) (2026)

## Abstract (arXiv)

True intelligence hinges on the ability to uncover and leverage hidden causal relations. Despite significant progress in AI and computer vision (CV), there remains a lack of benchmarks for assessing models' abilities to infer latent causality from complex visual data. In this paper, we introduce textscCausal3D, a novel and comprehensive benchmark that integrates structured data (tables) with corresponding visual representations (images) to evaluate causal reasoning. Designed within a systematic framework, Causal3D comprises 19 3D-scene datasets capturing diverse causal relations, views, and backgrounds, enabling evaluations across scenes of varying complexity. We assess multiple state-of-the-art methods, including classical causal discovery, causal representation learning, and large/vision-language models (LLMs/VLMs). Our experiments show that as causal structures grow more complex without prior knowledge, performance declines significantly, highlighting the challenges even advanced methods face in complex causal scenarios. Causal3D serves as a vital resource for advancing causal reasoning in CV and fostering trustworthy AI in critical domains.