# The Magic of IF: Investigating Causal Reasoning Abilities in Large Language Models of Code

arXiv: https://arxiv.org/abs/2305.19213 · HTML: https://mlsanity.com/paper/2305.19213/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning.
- Builds on 13 of its 38 references that are in the map.
- First posted on arXiv 30 May 2023.

## Placed in

- LLM reasoning / [Causal reasoning in LLMs](https://mlsanity.com/lr/causal-reasoning-in-llms/)

## Builds on (most cited first, 13 of 13)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [PAL: Program-aided Language Models](https://mlsanity.com/paper/2211.10435/) (2022)
- [Autoformalization with Large Language Models](https://mlsanity.com/paper/2205.12615/) (2022)
- [COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics](https://mlsanity.com/paper/2202.11705/) (2022)
- [Language Models of Code are Few-Shot Commonsense Learners](https://mlsanity.com/paper/2210.07128/) (2022)
- [Towards Zero-Label Language Learning](https://mlsanity.com/paper/2109.09193/) (2021)
- [Code4Struct: Code Generation for Few-Shot Event Structure Prediction](https://mlsanity.com/paper/2210.12810/) (2022)
- [CARE: Causality Reasoning for Empathetic Responses by Conditional Graph Generation](https://mlsanity.com/paper/2211.00255/) (2022)

## Built on by (most cited first, 21 of 21)

- [Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data](https://mlsanity.com/paper/2402.17644/) (2024)
- [LLaRA: Supercharging Robot Learning Data for Vision-Language Policy](https://mlsanity.com/paper/2406.20095/) (2024)
- [Large Language Models and Causal Inference in Collaboration: A Survey](https://mlsanity.com/paper/2403.09606/) (2024)
- [Understanding Long Videos with Multimodal Language Models](https://mlsanity.com/paper/2403.16998/) (2024)
- [Cause and Effect: Can Large Language Models Truly Understand Causality?](https://mlsanity.com/paper/2402.18139/) (2024)
- [When Do Program-of-Thoughts Work for Reasoning?](https://mlsanity.com/paper/2308.15452/) (2023)
- [Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches](https://mlsanity.com/paper/2501.03151/) (2025)
- [A Critical Review of Causal Reasoning Benchmarks for Large Language Models](https://mlsanity.com/paper/2407.08029/) (2024)
- [Counterfactual Token Generation in Large Language Models](https://mlsanity.com/paper/2409.17027/) (2024)
- [Event Causality Is Key to Computational Story Understanding](https://mlsanity.com/paper/2311.09648/) (2023)
- [MARS: Benchmarking the Metaphysical Reasoning Abilities of Language Models with a Multi-task Evaluation Dataset](https://mlsanity.com/paper/2406.02106/) (2024)
- [Applying Large Language Models for Causal Structure Learning in Non Small Cell Lung Cancer](https://mlsanity.com/paper/2311.07191/) (2023)
- [Can LLMs Correct Physicians, Yet? Investigating Effective Interaction Methods in the Medical Domain](https://mlsanity.com/paper/2403.20288/) (2024)
- [Code Execution as Grounded Supervision for LLM Reasoning](https://mlsanity.com/paper/2506.10343/) (2025)
- [Wiring the 'Why': A Unified Taxonomy and Survey of Abductive Reasoning in LLMs](https://mlsanity.com/paper/2604.08016/) (2026)
- [Do LLMs Have the Generalization Ability in Conducting Causal Inference?](https://mlsanity.com/paper/2410.11385/) (2024)
- [Do Large Language Models Show Biases in Causal Learning? Insights from Contingency Judgment](https://mlsanity.com/paper/2510.13985/) (2025)
- [Representation in large language models](https://mlsanity.com/paper/2501.00885/) (2025)
- [If Probable, Then Acceptable? Understanding Conditional Acceptability Judgments in Large Language Models](https://mlsanity.com/paper/2510.08388/) (2025)
- [Retell, Reward, Repeat: Reinforcement Learning for Narrative Theory-Informed Story Retelling](https://mlsanity.com/paper/2601.17226/) (2026)
- [Contextual Causality with Large Language Models: A Survey](https://mlsanity.com/paper/2609.22409/) (2026)

## Abstract (arXiv)

Causal reasoning, the ability to identify cause-and-effect relationship, is crucial in human thinking. Although large language models (LLMs) succeed in many NLP tasks, it is still challenging for them to conduct complex causal reasoning like abductive reasoning and counterfactual reasoning. Given the fact that programming code may express causal relations more often and explicitly with conditional statements like ``if``, we want to explore whether Code-LLMs acquire better causal reasoning abilities. Our experiments show that compared to text-only LLMs, Code-LLMs with code prompts are significantly better in causal reasoning. We further intervene on the prompts from different aspects, and discover that the programming structure is crucial in code prompt design, while Code-LLMs are robust towards format perturbations.