Probing Causality Manipulation of Large Language Models
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) have shown various ability on natural language processing, including problems about causality. It is not intuitive for LLMs to command causality, since pretrained models usually work on statistical associations, and do not focus on causes and effects in sentences. So that probing internal manipulation of causality is necessary for LLMs. This paper proposes a novel approach to probe causality manipulation hierarchically, by providing different shortcuts to models and observe behaviors. We exploit retrieval augmented generation (RAG) and in-context learning (ICL) for models on a designed causality classification task. We conduct experiments on mainstream LLMs, including GPT-4 and some smaller and domain-specific models. Our results suggest that LLMs can detect entities related to causality and recognize direct causal relationships. However, LLMs lack specialized cognition for causality, merely treating them as part of the global semantic of the sentence.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 21 of its 35 references that are in the map. First posted on arXiv 26 Aug 2024.
Builds on
21 of 35 references.
References · 21
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2017Billion-scale similarity search with GPUscited by 2,477
- 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
- 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
- 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
- 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
- 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
- 2019Abductive Commonsense Reasoningcited by 300
9 more
- 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
- 2022Predictability and Surprise in Large Generative Modelscited by 209
- 2023Causal Parrots: Large Language Models May Talk Causality But Are Not Causalcited by 146
- 2023Can Large Language Models Infer Causation from Correlation?cited by 142
- 2021CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmarkcited by 99
- 2023Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluationcited by 69
- 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
- 2023MISGENDERED: Limits of Large Language Models in Understanding Pronounscited by 43
- 2023Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledgecited by 32
Built on by
Citing papers per year: 2024: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024Large Language Models and Causal Inference in Collaboration: A Survey
- 2024CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs
- 2025Retrieval Augmented Generation based Large Language Models for Causality Mining
- 2023Enhancing Causal Reasoning in Large Language Models: A Causal Attribution Model for Precision Fine-Tuning
- 2025CausalRAG: Integrating Causal Graphs into Retrieval-Augmented Generation
- 2024Causal Inference with Large Language Model: A Survey
- 2023CLadder: Assessing Causal Reasoning in Language Models
- 2024A Critical Review of Causal Reasoning Benchmarks for Large Language Models