# Mitigating Spurious Correlations in LLMs via Causality-Aware Post-Training

arXiv: https://arxiv.org/abs/2506.09433 · HTML: https://mlsanity.com/paper/2506.09433/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 47 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Builds on (most cited first, 30 of 39)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning](https://mlsanity.com/paper/2305.12295/) (2023)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought](https://mlsanity.com/paper/2210.01240/) (2022)
- [ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language](https://mlsanity.com/paper/2012.13048/) (2020)
- [Transformers as Soft Reasoners over Language](https://mlsanity.com/paper/2002.05867/) (2020)
- [FOLIO: Natural Language Reasoning with First-Order Logic](https://mlsanity.com/paper/2209.00840/) (2022)
- [Large Language Models as Commonsense Knowledge for Large-Scale Task Planning](https://mlsanity.com/paper/2305.14078/) (2023)
- [Causal Parrots: Large Language Models May Talk Causality But Are Not Causal](https://mlsanity.com/paper/2308.13067/) (2023)
- [Can Large Language Models Infer Causation from Correlation?](https://mlsanity.com/paper/2306.05836/) (2023)
- [CLadder: Assessing Causal Reasoning in Language Models](https://mlsanity.com/paper/2312.04350/) (2023)
- [On the Paradox of Learning to Reason from Data](https://mlsanity.com/paper/2205.11502/) (2022)
- [Datasets for Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2402.18041/) (2024)
- [Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learning](https://mlsanity.com/paper/2305.17256/) (2023)
- [ZIN: When and How to Learn Invariance Without Environment Partition?](https://mlsanity.com/paper/2203.05818/) (2022)
- [Generating Natural Language Proofs with Verifier-Guided Search](https://mlsanity.com/paper/2205.12443/) (2022)
- [Scaling Laws of Synthetic Data for Language Models](https://mlsanity.com/paper/2503.19551/) (2025)

## Built on by (most cited first, 5 of 5)

- [CaRT: Teaching LLM Agents to Know When They Know Enough](https://mlsanity.com/paper/2510.08517/) (2025)
- [IslamicLegalBench: Evaluating LLMs Knowledge and Reasoning of Islamic Law Across 1,200 Years of Islamic Pluralist Legal Traditions](https://mlsanity.com/paper/2602.21226/) (2026)
- [Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck](https://mlsanity.com/paper/2603.10351/) (2026)
- [Supervised Fine-Tuning Needs to Unlock the Potential of Token Priority](https://mlsanity.com/paper/2602.01227/) (2026)
- [X-MULTI: VLM-based Imaging Factor Disentanglement for Factor-Aware Image Synthesis](https://mlsanity.com/paper/2608.24563/) (2026)

## Abstract (arXiv)

While large language models (LLMs) have demonstrated remarkable capabilities in language modeling, recent studies reveal that they often fail on out-of-distribution (OOD) samples due to spurious correlations acquired during pre-training. Here, we aim to mitigate such spurious correlations through causality-aware post-training (CAPT). By decomposing a biased prediction into two unbiased steps, known as textitevent estimation and textitevent intervention, we reduce LLMs' pre-training biases without incurring additional fine-tuning biases, thus enhancing the model's generalization ability. Experiments on the formal causal inference benchmark CLadder and the logical reasoning dataset PrOntoQA show that 3B-scale language models fine-tuned with CAPT can outperform both traditional SFT and larger LLMs on in-distribution (ID) and OOD tasks using only 100 ID fine-tuning samples, demonstrating the effectiveness and sample efficiency of CAPT.