# Context-Grounding Gains Are Mediated by Pre-existing Machinery: Auditing GRPO, SFT, and DPO

arXiv: https://arxiv.org/abs/2609.00925 · HTML: https://mlsanity.com/paper/2609.00925/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 18 of its 63 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 18 of 18)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [The Hydra Effect: Emergent Self-repair in Language Model Computations](https://mlsanity.com/paper/2307.15771/) (2023)
- [The Invisible Leash: Why RLVR May or May Not Escape Its Origin](https://mlsanity.com/paper/2507.14843/) (2025)
- [Towards Reliable Evaluation of Behavior Steering Interventions in LLMs](https://mlsanity.com/paper/2410.17245/) (2024)
- [Reasoning-Finetuning Repurposes Latent Representations in Base Models](https://mlsanity.com/paper/2507.12638/) (2025)
- [Evaluating GRPO and DPO for Faithful Chain-of-Thought Reasoning in LLMs](https://mlsanity.com/paper/2512.22631/) (2025)
- [EvidenceRL: Reinforcing Evidence Consistency for Trustworthy Language Models](https://mlsanity.com/paper/2603.19532/) (2026)
- [Better World Models Can Lead to Better Post-Training Performance](https://mlsanity.com/paper/2512.03400/) (2025)
- [ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models](https://mlsanity.com/paper/2601.04131/) (2026)
- [SHIFT: Gate-Modulated Activation Steering for Knowledge Conflict Mitigation in Retrieval-Augmented Generation](https://mlsanity.com/paper/2606.27786/) (2026)
- [Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models](https://mlsanity.com/paper/2606.28273/) (2026)
- [Emergent Misalignment Recruits a Pre-existing Persona Subspace](https://mlsanity.com/paper/2607.21356/) (2026)
- [Demystifying Variance in Circuit Discovery of LLMs](https://mlsanity.com/paper/2606.16920/) (2026)
- [Emergence of Context Characteristics Sensitivity in Large Language Models](https://mlsanity.com/paper/2606.09525/) (2026)
- [Actionable Activation Directions for Detecting and Mitigating Emergent Misalignment Across Language Model Families](https://mlsanity.com/paper/2606.20225/) (2026)
- [How Does Alignment Tuning Shape Representations of Sycophancy and Related Cue-Induced Biases in LLMs?](https://mlsanity.com/paper/2607.18114/) (2026)

## Abstract (arXiv)

Language models can ignore prompt evidence when it conflicts with memorized knowledge. Post-training can make models follow such evidence more reliably, but it is unclear whether these gains require new machinery or strengthen machinery already present. We compare nine post-training arms spanning GRPO, SFT, and DPO from one starting checkpoint, with key comparisons extended across scales and families. We estimate a grounding direction from that checkpoint before training. Across five tested GRPO variants, grounding gains are small. For the two variants replicated across seeds, equivalence tests bound their effects below the conflict-SFT gain even as the rewarded metric improves. Conflict-SFT improves grounding moderately, while DPO drives grounding near ceiling on its matched distribution. Conflict-SFT and DPO largely use the same causal attention-head set as the starting model. Subtracting the starting-model direction suppresses both gains, while adding it to the starting model recovers 35% of DPO's gain at a dose passing all stated side-effect checks. After a supervised warm start makes the context answer appear in more rollouts, the same GRPO recipe adds essentially no further grounding gain. In our setting, grounding gains largely depend on machinery already present in the starting model.