# Predicting Directionality in Causal Relations in Text

arXiv: https://arxiv.org/abs/2103.13606 · HTML: https://mlsanity.com/paper/2103.13606/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 41 references that are in the map.
- First posted on arXiv 25 Mar 2021.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [GLUCOSE: GeneraLized and COntextualized Story Explanations](https://mlsanity.com/paper/2009.07758/) (2020)
- [How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context](https://mlsanity.com/paper/2002.00652/) (2020)

## Built on by (most cited first, 8 of 8)

- [Causal Direction of Data Collection Matters: Implications of Causal and Anticausal Learning for NLP](https://mlsanity.com/paper/2110.03618/) (2021)
- [Discovering and Reasoning of Causality in the Hidden World with Large Language Models](https://mlsanity.com/paper/2402.03941/) (2024)
- [HeadlineCause: A Dataset of News Headlines for Detecting Causalities](https://mlsanity.com/paper/2108.12626/) (2021)
- [Causality from Bottom to Top: A Survey](https://mlsanity.com/paper/2403.11219/) (2024)
- [ExpliCa: Evaluating Explicit Causal Reasoning in Large Language Models](https://mlsanity.com/paper/2502.15487/) (2025)
- [Investigating Counterclaims in Causality Extraction from Text](https://mlsanity.com/paper/2510.08224/) (2025)
- [A Cross-Domain Evaluation of Approaches for Causal Knowledge Extraction](https://mlsanity.com/paper/2308.03891/) (2023)
- [Fast and Generalizable NeRF Architecture Selection for Satellite Scene Reconstruction](https://mlsanity.com/paper/2603.18306/) (2026)

## Abstract (arXiv)

In this work, we test the performance of two bidirectional transformer-based language models, BERT and SpanBERT, on predicting directionality in causal pairs in the textual content. Our preliminary results show that predicting direction for inter-sentence and implicit causal relations is more challenging. And, SpanBERT performs better than BERT on causal samples with longer span length. We also introduce CREST which is a framework for unifying a collection of scattered datasets of causal relations.