# Is ChatGPT the Future of Causal Text Mining? A Comprehensive Evaluation and Analysis

arXiv: https://arxiv.org/abs/2402.14484 · HTML: https://mlsanity.com/paper/2402.14484/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 61 references that are in the map.
- First posted on arXiv 22 Feb 2024.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [Sentiment Analysis in the Era of Large Language Models: A Reality Check](https://mlsanity.com/paper/2305.15005/) (2023)
- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [Is ChatGPT a Good Sentiment Analyzer? A Preliminary Study](https://mlsanity.com/paper/2304.04339/) (2023)
- [Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation](https://mlsanity.com/paper/2305.07375/) (2023)
- [TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks](https://mlsanity.com/paper/2305.11430/) (2023)
- [ChatGPT Evaluation on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relations](https://mlsanity.com/paper/2304.14827/) (2023)
- [Causal-Discovery Performance of ChatGPT in the context of Neuropathic Pain Diagnosis](https://mlsanity.com/paper/2301.13819/) (2023)
- [Guided Generation of Cause and Effect](https://mlsanity.com/paper/2107.09846/) (2021)
- [To Test Machine Comprehension, Start by Defining Comprehension](https://mlsanity.com/paper/2005.01525/) (2020)
- [Financial Document Causality Detection Shared Task (FinCausal 2020)](https://mlsanity.com/paper/2012.02505/) (2020)
- [A Survey on Extraction of Causal Relations from Natural Language Text](https://mlsanity.com/paper/2101.06426/) (2021)
- [The Causal News Corpus: Annotating Causal Relations in Event Sentences from News](https://mlsanity.com/paper/2204.11714/) (2022)
- [On the Importance of Data Size in Probing Fine-tuned Models](https://mlsanity.com/paper/2203.09627/) (2022)
- [UniCausal: Unified Benchmark and Repository for Causal Text Mining](https://mlsanity.com/paper/2208.09163/) (2022)

## Built on by (most cited first, 2 of 2)

- [Benchmarking LLMs for Pairwise Causal Discovery in Biomedical and Multi-Domain Contexts](https://mlsanity.com/paper/2601.15479/) (2026)
- [Identifying attributions of causality in political text](https://mlsanity.com/paper/2512.03214/) (2025)

## Abstract (arXiv)

Causality is fundamental in human cognition and has drawn attention in diverse research fields. With growing volumes of textual data, discerning causalities within text data is crucial, and causal text mining plays a pivotal role in extracting meaningful patterns. This study conducts comprehensive evaluations of ChatGPT's causal text mining capabilities. Firstly, we introduce a benchmark that extends beyond general English datasets, including domain-specific and non-English datasets. We also provide an evaluation framework to ensure fair comparisons between ChatGPT and previous approaches. Finally, our analysis outlines the limitations and future challenges in employing ChatGPT for causal text mining. Specifically, our analysis reveals that ChatGPT serves as a good starting point for various datasets. However, when equipped with a sufficient amount of training data, previous models still surpass ChatGPT's performance. Additionally, ChatGPT suffers from the tendency to falsely recognize non-causal sequences as causal sequences. These issues become even more pronounced with advanced versions of the model, such as GPT-4. In addition, we highlight the constraints of ChatGPT in handling complex causality types, including both intra/inter-sentential and implicit causality. The model also faces challenges with effectively leveraging in-context learning and domain adaptation. We release our code to support further research and development in this field.