mlsanityData to 7 Oct 2026

The Magic of IF: Investigating Causal Reasoning Abilities in Large Language Models of Code

arXiv 2305.19213 ↗First posted 30 May 2023cs.CLcited by 21

On the map: LLM reasoning → Causal reasoning in LLMs

Abstract (arXiv)

Causal reasoning, the ability to identify cause-and-effect relationship, is crucial in human thinking. Although large language models (LLMs) succeed in many NLP tasks, it is still challenging for them to conduct complex causal reasoning like abductive reasoning and counterfactual reasoning. Given the fact that programming code may express causal relations more often and explicitly with conditional statements like ``if``, we want to explore whether Code-LLMs acquire better causal reasoning abilities. Our experiments show that compared to text-only LLMs, Code-LLMs with code prompts are significantly better in causal reasoning. We further intervene on the prompts from different aspects, and discover that the programming structure is crucial in code prompt design, while Code-LLMs are robust towards format perturbations.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning. Builds on 13 of its 38 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

13 of 38 references.

References · 13

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  4. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  5. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  6. 2023StarCoder: may the source be with you!cited by 790
  7. 2022PAL: Program-aided Language Modelscited by 635
  8. 2022Autoformalization with Large Language Modelscited by 252
  9. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  10. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  11. 2021Towards Zero-Label Language Learningcited by 76
  12. 2022Code4Struct: Code Generation for Few-Shot Event Structure Predictioncited by 70
1 more
  1. 2022CARE: Causality Reasoning for Empathetic Responses by Conditional Graph Generationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 10; 2025: 5; 2026: 3.

21 papers.

Sort:

By area:

  1. 2024Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Datacited by 61
  2. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 51
  3. 2024Large Language Models and Causal Inference in Collaboration: A Surveycited by 26
  4. 2024Understanding Long Videos with Multimodal Language Modelscited by 25
  5. 2024Cause and Effect: Can Large Language Models Truly Understand Causality?cited by 23
  6. 2023When Do Program-of-Thoughts Work for Reasoning?cited by 19
  7. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  8. 2024Counterfactual Token Generation in Large Language Modelscited by 14
  9. 2024A Critical Review of Causal Reasoning Benchmarks for Large Language Modelscited by 13
  10. 2023Event Causality Is Key to Computational Story Understandingcited by 11
  11. 2024MARS: Benchmarking the Metaphysical Reasoning Abilities of Language Models with a Multi-task Evaluation Datasetcited by 11
  12. 2023Applying Large Language Models for Causal Structure Learning in Non Small Cell Lung Cancercited by 8
9 more
  1. 2024Can LLMs Correct Physicians, Yet? Investigating Effective Interaction Methods in the Medical Domaincited by 7
  2. 2025Code Execution as Grounded Supervision for LLM Reasoningcited by 5
  3. 2026Wiring the 'Why': A Unified Taxonomy and Survey of Abductive Reasoning in LLMscited by 5
  4. 2024Do LLMs Have the Generalization Ability in Conducting Causal Inference?cited by 2
  5. 2025Do Large Language Models Show Biases in Causal Learning? Insights from Contingency Judgmentcited by 1
  6. 2025Representation in large language modelscited by 0
  7. 2025If Probable, Then Acceptable? Understanding Conditional Acceptability Judgments in Large Language Modelscited by 0
  8. 2026Retell, Reward, Repeat: Reinforcement Learning for Narrative Theory-Informed Story Retellingcited by 0
  9. 2026Contextual Causality with Large Language Models: A Surveycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs
  2. 2025Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
  3. 2025Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment
  4. 2025A Survey on Enhancing Causal Reasoning Ability of Large Language Models
  5. 2025On Code-Induced Reasoning in LLMs
  6. 2024Eliciting Causal Abilities in Large Language Models for Reasoning Tasks
  7. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causality
  8. 2023CLadder: Assessing Causal Reasoning in Language Models