# CELLO: Causal Evaluation of Large Vision-Language Models

arXiv: https://arxiv.org/abs/2406.19131 · HTML: https://mlsanity.com/paper/2406.19131/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 34 of its 74 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 34)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback](https://mlsanity.com/paper/2312.00849/) (2023)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [Visual7W: Grounded Question Answering in Images](https://mlsanity.com/paper/1511.03416/) (2015)
- [FVQA: Fact-based Visual Question Answering](https://mlsanity.com/paper/1606.05433/) (2016)
- [TVQA+: Spatio-Temporal Grounding for Video Question Answering](https://mlsanity.com/paper/1904.11574/) (2019)
- [Causal Parrots: Large Language Models May Talk Causality But Are Not Causal](https://mlsanity.com/paper/2308.13067/) (2023)
- [Can Large Language Models Infer Causation from Correlation?](https://mlsanity.com/paper/2306.05836/) (2023)
- [VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena](https://mlsanity.com/paper/2112.07566/) (2021)
- [CLadder: Assessing Causal Reasoning in Language Models](https://mlsanity.com/paper/2312.04350/) (2023)
- [Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning](https://mlsanity.com/paper/1811.02307/) (2018)
- [Discovering Causal Signals in Images](https://mlsanity.com/paper/1605.08179/) (2016)
- [Reasoning about Goals, Steps, and Temporal Ordering with WikiHow](https://mlsanity.com/paper/2009.07690/) (2020)
- [IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation](https://mlsanity.com/paper/2104.08200/) (2021)
- [Understanding Causality with Large Language Models: Feasibility and Opportunities](https://mlsanity.com/paper/2304.05524/) (2023)
- [Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages](https://mlsanity.com/paper/2308.12038/) (2023)
- [Overcoming Language Priors with Self-supervised Learning for Visual Question Answering](https://mlsanity.com/paper/2012.11528/) (2020)
- [Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective](https://mlsanity.com/paper/2403.18346/) (2024)
- [PaperRobot: Incremental Draft Generation of Scientific Ideas](https://mlsanity.com/paper/1905.07870/) (2019)
- [The Essential Role of Causality in Foundation World Models for Embodied AI](https://mlsanity.com/paper/2402.06665/) (2024)

## Built on by (most cited first, 10 of 10)

- [MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs](https://mlsanity.com/paper/2411.15296/) (2024)
- [CF-VLM:CounterFactual Vision-Language Fine-tuning](https://mlsanity.com/paper/2506.17267/) (2025)
- [Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision](https://mlsanity.com/paper/2504.02477/) (2025)
- [Graph is a Substrate Across Data Modalities](https://mlsanity.com/paper/2601.22384/) (2026)
- [Retrieving Counterfactuals Improves Visual In-Context Learning](https://mlsanity.com/paper/2603.16737/) (2026)
- [Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs](https://mlsanity.com/paper/2606.05966/) (2026)
- [CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Models](https://mlsanity.com/paper/2412.17970/) (2024)
- [Discovering and using Spelke segments](https://mlsanity.com/paper/2507.16038/) (2025)
- [Distilling Counterfactual Reasoning from Language to Vision: Causal Graph Guided Post-Training for Video Understanding](https://mlsanity.com/paper/2511.19923/) (2025)
- [CauSight: Learning to Supersense for Visual Causal Discovery](https://mlsanity.com/paper/2512.01827/) (2025)

## Abstract (arXiv)

Causal reasoning is fundamental to human intelligence and crucial for effective decision-making in real-world environments. Despite recent advancements in large vision-language models (LVLMs), their ability to comprehend causality remains unclear. Previous work typically focuses on commonsense causality between events and/or actions, which is insufficient for applications like embodied agents and lacks the explicitly defined causal graphs required for formal causal reasoning. To overcome these limitations, we introduce a fine-grained and unified definition of causality involving interactions between humans and/or objects. Building on the definition, we construct a novel dataset, CELLO, consisting of 14,094 causal questions across all four levels of causality: discovery, association, intervention, and counterfactual. This dataset surpasses traditional commonsense causality by including explicit causal graphs that detail the interactions between humans and objects. Extensive experiments on CELLO reveal that current LVLMs still struggle with causal reasoning tasks, but they can benefit significantly from our proposed CELLO-CoT, a causally inspired chain-of-thought prompting strategy. Both quantitative and qualitative analyses from this study provide valuable insights for future research. Our project page is at https://github.com/OpenCausaLab/CELLO.