# Causal Distillation: Transferring Structured Explanations from Large to Compact Language Models

arXiv: https://arxiv.org/abs/2505.19511 · HTML: https://mlsanity.com/paper/2505.19511/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning.
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 26 May 2025.

## Placed in

- LLM reasoning / [Causal reasoning in LLMs](https://mlsanity.com/lr/causal-reasoning-in-llms/)

## Builds on (most cited first, 19 of 19)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Teaching Small Language Models to Reason](https://mlsanity.com/paper/2212.08410/) (2022)
- [CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims](https://mlsanity.com/paper/2012.00614/) (2020)
- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [Phi-4-reasoning Technical Report](https://mlsanity.com/paper/2504.21318/) (2025)
- [Can Large Language Models Infer Causation from Correlation?](https://mlsanity.com/paper/2306.05836/) (2023)
- [A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness](https://mlsanity.com/paper/2411.03350/) (2024)
- [CLadder: Assessing Causal Reasoning in Language Models](https://mlsanity.com/paper/2312.04350/) (2023)
- [Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?](https://mlsanity.com/paper/2010.04119/) (2020)
- [Understanding Causality with Large Language Models: Feasibility and Opportunities](https://mlsanity.com/paper/2304.05524/) (2023)
- [Cause and Effect: Can Large Language Models Truly Understand Causality?](https://mlsanity.com/paper/2402.18139/) (2024)
- [Causal Distillation for Language Models](https://mlsanity.com/paper/2112.02505/) (2021)
- [Causal Evaluation of Language Models](https://mlsanity.com/paper/2405.00622/) (2024)
- [Enhancing Causal Reasoning in Large Language Models: A Causal Attribution Model for Precision Fine-Tuning](https://mlsanity.com/paper/2401.00139/) (2023)
- [Are Human Explanations Always Helpful? Towards Objective Evaluation of Human Natural Language Explanations](https://mlsanity.com/paper/2305.03117/) (2023)
- [A Survey on Enhancing Causal Reasoning Ability of Large Language Models](https://mlsanity.com/paper/2503.09326/) (2025)

## Built on by (most cited first, 2 of 2)

- [A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthiness](https://mlsanity.com/paper/2510.13890/) (2025)
- [CausalOPD: First-Wrong-Step Supervision for Distilling Causal Chain Reasoning](https://mlsanity.com/paper/2608.03673/) (2026)

## Abstract (arXiv)

Large proprietary language models exhibit strong causal reasoning abilities that smaller open-source models struggle to replicate. We introduce a novel framework for distilling causal explanations that transfers causal reasoning skills from a powerful teacher model to a compact open-source model. The key idea is to train the smaller model to develop causal reasoning abilities by generating structured cause-and-effect explanations consistent with those of the teacher model. To evaluate the quality of the student-generated explanations, we introduce a new metric called Causal Explanation Coherence (CEC) to assess the structural and logical consistency of causal reasoning. This metric uses sentence-level semantic alignment to measure how well each part of the generated explanation corresponds to the teacher's reference, capturing both faithfulness and coverage of the underlying causal chain. Our framework and the CEC metric provide a principled foundation for training smaller models to perform robust causal reasoning and for systematically assessing the coherence of explanations in language model outputs.