# The Impact of Post-training on Data Contamination

arXiv: https://arxiv.org/abs/2601.06103 · HTML: https://mlsanity.com/paper/2601.06103/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 3 Jan 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 18 of 18)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models](https://mlsanity.com/paper/2504.11468/) (2025)
- [OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data](https://mlsanity.com/paper/2410.01560/) (2024)
- [Rethinking Benchmark and Contamination for Language Models with Rephrased Samples](https://mlsanity.com/paper/2311.04850/) (2023)
- [Time Travel in LLMs: Tracing Data Contamination in Large Language Models](https://mlsanity.com/paper/2308.08493/) (2023)
- [GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers](https://mlsanity.com/paper/2402.19255/) (2024)
- [Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks](https://mlsanity.com/paper/2305.10160/) (2023)
- [Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models](https://mlsanity.com/paper/2402.15938/) (2024)
- [Investigating Data Contamination for Pre-training Language Models](https://mlsanity.com/paper/2401.06059/) (2024)
- [A Survey on Data Contamination for Large Language Models](https://mlsanity.com/paper/2502.14425/) (2025)
- [Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?](https://mlsanity.com/paper/2411.03923/) (2024)
- [Data Contamination Report from the 2024 CONDA Shared Task](https://mlsanity.com/paper/2407.21530/) (2024)
- [Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translation](https://mlsanity.com/paper/2501.18771/) (2025)

## Built on by (most cited first, 5 of 5)

- [Quantifying the Effect of Test Set Contamination on Generative Evaluations](https://mlsanity.com/paper/2601.04301/) (2026)
- [TEMPO: Temporal Enforcement via Mode-Separated Policy Optimization for Trustworthy LLM Backtesting](https://mlsanity.com/paper/2605.18843/) (2026)
- [Trust Functions: Near-Lossless Weak-to-Strong Generalization by Learning When to Trust the Weak Teacher](https://mlsanity.com/paper/2606.01000/) (2026)
- [To Reason or to Fabricate: Reasoning Without Shortcuts via Hint-Anchored Pairwise Aggregation](https://mlsanity.com/paper/2606.29481/) (2026)
- [Auditing Future-Memory Use in Point-in-Time Language Agents](https://mlsanity.com/paper/2610.18602/) (2026)

## Abstract (arXiv)

We present a controlled study of how dataset contamination interacts with the post-training stages now standard in large language model training pipelines. Starting from clean checkpoints of Qwen2.5 (0.5B/1.5B) and Gemma3 (1B/4B), we inject five copies of GSM8K and MBPP test items into the first 2B tokens of an otherwise 25B token extended pre-training dataset. We then compare the contaminated and clean models both immediately after pre-training and again after two popular post-training methods: supervised fine-tuning (SFT) and reinforcement learning (RL) with group relative policy optimization (GRPO). The applied post-training steps do not have any contamination. Across math and coding benchmarks, we find three consistent patterns: (i) Contamination causes performance spikes that are gradually diminished with continued pre-training. After even 25B tokens the apparent performance inflation of contamination can become close to zero. (ii) Both SFT and GRPO resurface the leaked information, but with different external validity: SFT inflates scores only on the contaminated tasks, whereas GRPO also inflates performance on uncontaminated counterparts (GSMPlus, HumanEval). (iii) Model scale amplifies these tendencies, larger Supervised Fine Tuned models memorize more, while larger GRPO models translate leakage into more generalizable capabilities. Our results underscore the need for contamination audits after post-training and suggest that RL-based post-training, although not immune, can help alleviate contamination-related over-estimation problems.