mlsanityData to 7 Oct 2026

Can Large Language Models Learn Independent Causal Mechanisms?

arXiv 2402.02636 ↗First posted 4 Feb 2024cs.CL cs.AI cs.ITcited by 3

On the map: LLM reasoning → Causal reasoning in LLMs

Abstract (arXiv)

Despite impressive performance on language modelling and complex reasoning tasks, Large Language Models (LLMs) fall short on the same tasks in uncommon settings or with distribution shifts, exhibiting a lack of generalisation ability. By contrast, systems such as causal models, that learn abstract variables and causal relationships, can demonstrate increased robustness against changes in the distribution. One reason for this success is the existence and use of Independent Causal Mechanisms (ICMs) representing high-level concepts that only sparsely interact. In this work, we apply two concepts from causality to learn ICMs within LLMs. We develop a new LLM architecture composed of multiple sparsely interacting language modelling modules. We show that such causal constraints can improve out-of-distribution performance on abstract and causal reasoning tasks. We also investigate the level of independence and domain specialisation and show that LLMs rely on pre-trained partially domain-invariant mechanisms resilient to fine-tuning.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning. Builds on 32 of its 54 references that are in the map. First posted on arXiv 4 Feb 2024.

Builds on

32 of 54 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
20 more
  1. 2024Mixtral of Expertscited by 1,785
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  3. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 273
  4. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  5. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  6. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 225
  7. 2019Recurrent Independent Mechanismscited by 219
  8. 2021Towards Causal Representation Learningcited by 207
  9. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  10. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  11. 2023Causal Parrots: Large Language Models May Talk Causality But Are Not Causalcited by 146
  12. 2023Can Large Language Models Infer Causation from Correlation?cited by 142
  13. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  14. 2022Causal Transformer for Estimating Counterfactual Outcomescited by 121
  15. 2021Causal Attention for Vision-Language Taskscited by 111
  16. 2023SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMscited by 54
  17. 2022Is a Modular Architecture Enough?cited by 35
  18. 2021ACRE: Abstract Causal REasoning Beyond Covariationcited by 30
  19. 2021Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalizationcited by 29
  20. 2023A Survey of Methods, Challenges and Perspectives in Causalitycited by 6

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025A Survey on Enhancing Causal Reasoning Ability of Large Language Modelscited by 8
  2. 2025Pitfalls of Scale: Investigating the Inverse Task of Redefinition in Large Language Modelscited by 1
  3. 2025Causal Cartographer: From Mapping to Reasoning Over Counterfactual Worldscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
  2. 2024CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs
  3. 2024Large Language Models and Causal Inference in Collaboration: A Survey
  4. 2025Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learning
  5. 2024CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Models
  6. 2024C2P: Featuring Large Language Models with Causal Reasoning
  7. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causality
  8. 2023CLadder: Assessing Causal Reasoning in Language Models