# Teaching Transformers Causal Reasoning through Axiomatic Training

arXiv: https://arxiv.org/abs/2407.07612 · HTML: https://mlsanity.com/paper/2407.07612/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning.
- Builds on 3 of its 39 references that are in the map.
- First posted on arXiv 10 Jul 2024.

## Placed in

- LLM reasoning / [Causal reasoning in LLMs](https://mlsanity.com/lr/causal-reasoning-in-llms/)

## Builds on (most cited first, 3 of 3)

- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [Integrating Large Language Model for Improved Causal Discovery](https://mlsanity.com/paper/2306.16902/) (2023)

## Built on by (most cited first, 12 of 12)

- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [Executable Counterfactuals: Improving LLMs' Causal Reasoning Through Code](https://mlsanity.com/paper/2510.01539/) (2025)
- [Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning](https://mlsanity.com/paper/2507.10624/) (2025)
- [CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models](https://mlsanity.com/paper/2506.11034/) (2025)
- [CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists](https://mlsanity.com/paper/2605.26029/) (2026)
- [PropLLM: Propagation-Aware Scene Reconstruction for Network Fault Diagnosis](https://mlsanity.com/paper/2606.00582/) (2026)
- [Transformers Can Learn Connectivity in Some Graphs but Not Others](https://mlsanity.com/paper/2509.22343/) (2025)
- [On Semantic Loss Fine-Tuning Approach for Preventing Model Collapse in Causal Reasoning](https://mlsanity.com/paper/2605.05438/) (2026)
- [ORCA: An End-to-End Interactive Copilot for Optimized Root Cause Analysis](https://mlsanity.com/paper/2605.27022/) (2026)
- [Learning to Deny: Action Denial in Multimodal Large Language Models](https://mlsanity.com/paper/2606.31187/) (2026)
- [CausalSteward: An Agentic Divide-Conquer-Combine Copilot for Causal Discovery](https://mlsanity.com/paper/2607.01936/) (2026)
- [Evidence-Type Competition: When Can Interventional Data Teach Language Models Causal Direction?](https://mlsanity.com/paper/2607.29484/) (2026)

## Abstract (arXiv)

For text-based AI systems to interact in the real world, causal reasoning is an essential skill. Since active interventions are costly, we study to what extent a system can learn causal reasoning from symbolic demonstrations of causal axioms. Specifically, we present an axiomatic training method where the system learns from multiple demonstrations of a causal axiom (or rule), rather than incorporating the axiom as an inductive bias or inferring it from data values. A key question is whether the system would learn to generalize from the axiom demonstrations to more complex scenarios. Our results, based on applying axiomatic training to learn the transitivity axiom and d-separation rule, indicate that such generalization is possible. To avoid data contamination issues, we start with a 67 million parameter transformer model and train it from scratch. On both tasks, we find that a model trained on linear causal chains (along with some noisy variations) can generalize well to complex graphs, including longer causal chains, causal chains with reversed order, and graphs with branching.To handle diverse text inputs, the same method is extended to finetune language models. Finetuning Llama-3-8B-Instruct model on our axiomatic data leads to significant gains on causal benchmarks such as Corr2Cause and CLEAR, in some cases providing state-of-the-art performance surpassing GPT-4.