mlsanityData to 7 Oct 2026

Mitigating Spurious Correlations in LLMs via Causality-Aware Post-Training

arXiv 2506.09433 ↗First posted 11 Jun 2025cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

While large language models (LLMs) have demonstrated remarkable capabilities in language modeling, recent studies reveal that they often fail on out-of-distribution (OOD) samples due to spurious correlations acquired during pre-training. Here, we aim to mitigate such spurious correlations through causality-aware post-training (CAPT). By decomposing a biased prediction into two unbiased steps, known as textitevent estimation and textitevent intervention, we reduce LLMs' pre-training biases without incurring additional fine-tuning biases, thus enhancing the model's generalization ability. Experiments on the formal causal inference benchmark CLadder and the logical reasoning dataset PrOntoQA show that 3B-scale language models fine-tuned with CAPT can outperform both traditional SFT and larger LLMs on in-distribution (ID) and OOD tasks using only 100 ID fine-tuning samples, demonstrating the effectiveness and sample efficiency of CAPT.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 39 of its 47 references that are in the map. First posted on arXiv 11 Jun 2025.

Builds on

39 of 47 references.

References · 39

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2024Qwen2.5 Technical Reportcited by 4,713
  10. 2019Invariant Risk Minimizationcited by 1,832
  11. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  12. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
27 more
  1. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 533
  2. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  3. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 379
  4. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  5. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  6. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  7. 2020Transformers as Soft Reasoners over Languagecited by 265
  8. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 233
  9. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  10. 2023Causal Parrots: Large Language Models May Talk Causality But Are Not Causalcited by 146
  11. 2023Can Large Language Models Infer Causation from Correlation?cited by 142
  12. 2023CLadder: Assessing Causal Reasoning in Language Modelscited by 102
  13. 2022On the Paradox of Learning to Reason from Datacited by 93
  14. 2023Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learningcited by 69
  15. 2024Datasets for Large Language Models: A Comprehensive Surveycited by 69
  16. 2022ZIN: When and How to Learn Invariance Without Environment Partition?cited by 58
  17. 2022Generating Natural Language Proofs with Verifier-Guided Searchcited by 52
  18. 2025Scaling Laws of Synthetic Data for Language Modelscited by 44
  19. 2024The Clever Hans Mirage: A Comprehensive Survey on Spurious Correlations in Machine Learningcited by 39
  20. 2022Entailer: Answering Questions with Faithful and Truthful Chains of Reasoningcited by 38
  21. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Modelscited by 35
  22. 2022Modeling the Data-Generating Process is Necessary for Out-of-Distribution Generalizationcited by 34
  23. 2024Efficient Tool Use with Chain-of-Abstraction Reasoningcited by 33
  24. 2021Annotation Inconsistency and Entity Bias in MultiWOZcited by 30
  25. 2023A Causal View of Entity Bias in (Large) Language Modelscited by 21
  26. 2025Evaluating the Generalization Capabilities of Large Language Models on Code Reasoningcited by 7
  27. 2024Do LLMs Have the Generalization Ability in Conducting Causal Inference?cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025CaRT: Teaching LLM Agents to Know When They Know Enoughcited by 5
  2. 2026IslamicLegalBench: Evaluating LLMs Knowledge and Reasoning of Islamic Law Across 1,200 Years of Islamic Pluralist Legal Traditionscited by 5
  3. 2026Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneckcited by 2
  4. 2026Supervised Fine-Tuning Needs to Unlock the Potential of Token Prioritycited by 0
  5. 2026X-MULTI: VLM-based Imaging Factor Disentanglement for Factor-Aware Image Synthesiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Can Post-Training Transform LLMs into Causal Reasoners?
  2. 2024Large Language Models and Causal Inference in Collaboration: A Survey
  3. 2024CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs
  4. 2026CausalMix: Data Mixture as Causal Inference for Language Model Training
  5. 2023LLM4Causal: Democratized Causal Tools for Everyone via Large Language Model
  6. 2023Enhancing Causal Reasoning in Large Language Models: A Causal Attribution Model for Precision Fine-Tuning
  7. 2025Ice Cream Doesn't Cause Drowning: Benchmarking LLMs Against Statistical Pitfalls in Causal Inference
  8. 2026Causal methods for LLM development and evaluation