# Causal BERT : Language models for causality detection between events expressed in text

arXiv: https://arxiv.org/abs/2012.05453 · HTML: https://mlsanity.com/paper/2012.05453/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2020.

## Built on by (most cited first, 22 of 22)

- [Causal Parrots: Large Language Models May Talk Causality But Are Not Causal](https://mlsanity.com/paper/2308.13067/) (2023)
- [Integrating Large Language Models in Causal Discovery: A Statistical Causal Approach](https://mlsanity.com/paper/2402.01454/) (2024)
- [Can Foundation Models Talk Causality?](https://mlsanity.com/paper/2206.10591/) (2022)
- [Large Language Models are Effective Priors for Causal Graph Discovery](https://mlsanity.com/paper/2405.13551/) (2024)
- [Can LLMs Help You at Work? A Sandbox for Evaluating LLM Agents in Enterprise Environments](https://mlsanity.com/paper/2510.27287/) (2025)
- [MIMICause: Representation and automatic extraction of causal relation types from clinical notes](https://mlsanity.com/paper/2110.07090/) (2021)
- [Event Causality Is Key to Computational Story Understanding](https://mlsanity.com/paper/2311.09648/) (2023)
- [Causality extraction from medical text using Large Language Models (LLMs)](https://mlsanity.com/paper/2407.10020/) (2024)
- [Leveraging Prompt-Based Large Language Models: Predicting Pandemic Health Decisions and Outcomes Through Social Media Language](https://mlsanity.com/paper/2403.00994/) (2024)
- [A Survey of Event Causality Identification: Taxonomy, Challenges, Assessment, and Prospects](https://mlsanity.com/paper/2411.10371/) (2024)
- [A Survey of Online Hate Speech through the Causal Lens](https://mlsanity.com/paper/2109.08120/) (2021)
- [Investigating Counterclaims in Causality Extraction from Text](https://mlsanity.com/paper/2510.08224/) (2025)
- [WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoning](https://mlsanity.com/paper/2606.11816/) (2026)
- [Identifying causal relations in tweets using deep learning: Use case on diabetes-related tweets from 2017-2021](https://mlsanity.com/paper/2111.01225/) (2021)
- [IDIAPers @ Causal News Corpus 2022: Efficient Causal Relation Identification Through a Prompt-based Few-shot Approach](https://mlsanity.com/paper/2209.03895/) (2022)
- [Machine Learning-based NLP for Emotion Classification on a Cholera X Dataset](https://mlsanity.com/paper/2405.04897/) (2024)
- [An Empirical Study of Causal Relation Extraction Transfer: Design and Data](https://mlsanity.com/paper/2503.06076/) (2025)
- [Powering Effective Climate Communication with a Climate Knowledge Base](https://mlsanity.com/paper/2107.11351/) (2021)
- [Self-Labeling in Multivariate Causality and Quantification for Adaptive Machine Learning](https://mlsanity.com/paper/2404.05809/) (2024)
- [Increasing the Accessibility of Causal Domain Knowledge via Causal Information Extraction Methods: A Case Study in the Semiconductor Manufacturing Industry](https://mlsanity.com/paper/2411.10172/) (2024)
- [Evaluating Large Language Models for Causal Modeling](https://mlsanity.com/paper/2411.15888/) (2024)
- [Causal Cartographer: From Mapping to Reasoning Over Counterfactual Worlds](https://mlsanity.com/paper/2505.14396/) (2025)

## Abstract (arXiv)

Causality understanding between events is a critical natural language processing task that is helpful in many areas, including health care, business risk management and finance. On close examination, one can find a huge amount of textual content both in the form of formal documents or in content arising from social media like Twitter, dedicated to communicating and exploring various types of causality in the real world. Recognizing these "Cause-Effect" relationships between natural language events continues to remain a challenge simply because it is often expressed implicitly. Implicit causality is hard to detect through most of the techniques employed in literature and can also, at times be perceived as ambiguous or vague. Also, although well-known datasets do exist for this problem, the examples in them are limited in the range and complexity of the causal relationships they depict especially when related to implicit relationships. Most of the contemporary methods are either based on lexico-semantic pattern matching or are feature-driven supervised methods. Therefore, as expected these methods are more geared towards handling explicit causal relationships leading to limited coverage for implicit relationships and are hard to generalize. In this paper, we investigate the language model's capabilities for causal association among events expressed in natural language text using sentence context combined with event information, and by leveraging masked event context with in-domain and out-of-domain data distribution. Our proposed methods achieve the state-of-art performance in three different data distributions and can be leveraged for extraction of a causal diagram and/or building a chain of events from unstructured text.