# Learning Contextual Causality from Time-consecutive Images

arXiv: https://arxiv.org/abs/2012.07138 · HTML: https://mlsanity.com/paper/2012.07138/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2020.

## Built on by (most cited first, 4 of 4)

- [The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning](https://mlsanity.com/paper/2202.04800/) (2022)
- [WhyAct: Identifying Action Reasons in Lifestyle Vlogs](https://mlsanity.com/paper/2109.02747/) (2021)
- [Finding Differences Between Transformers and ConvNets Using Counterfactual Simulation Testing](https://mlsanity.com/paper/2211.16499/) (2022)
- [iReason: Multimodal Commonsense Reasoning using Videos and Natural Language with Interpretability](https://mlsanity.com/paper/2107.10300/) (2021)

## Abstract (arXiv)

Causality knowledge is crucial for many artificial intelligence systems. Conventional textual-based causality knowledge acquisition methods typically require laborious and expensive human annotations. As a result, their scale is often limited. Moreover, as no context is provided during the annotation, the resulting causality knowledge records (e.g., ConceptNet) typically do not take the context into consideration. To explore a more scalable way of acquiring causality knowledge, in this paper, we jump out of the textual domain and investigate the possibility of learning contextual causality from the visual signal. Compared with pure text-based approaches, learning causality from the visual signal has the following advantages: (1) Causality knowledge belongs to the commonsense knowledge, which is rarely expressed in the text but rich in videos; (2) Most events in the video are naturally time-ordered, which provides a rich resource for us to mine causality knowledge from; (3) All the objects in the video can be used as context to study the contextual property of causal relations. In detail, we first propose a high-quality dataset Vis-Causal and then conduct experiments to demonstrate that with good language and visual representation models as well as enough training signals, it is possible to automatically discover meaningful causal knowledge from the videos. Further analysis also shows that the contextual property of causal relations indeed exists, taking which into consideration might be crucial if we want to use the causality knowledge in real applications, and the visual signal could serve as a good resource for learning such contextual causality.