# Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination

arXiv: https://arxiv.org/abs/2507.10532 · HTML: https://mlsanity.com/paper/2507.10532/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 29 of its 35 references that are in the map.
- First posted on arXiv 14 Jul 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 29 of 29)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [Absolute Zero: Reinforced Self-play Reasoning with Zero Data](https://mlsanity.com/paper/2505.03335/) (2025)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model](https://mlsanity.com/paper/2502.02737/) (2025)
- [Reinforcement Learning for Reasoning in Large Language Models with One Training Example](https://mlsanity.com/paper/2504.20571/) (2025)
- [TTRL: Test-Time Reinforcement Learning](https://mlsanity.com/paper/2504.16084/) (2025)
- [Learning to Reason without External Rewards](https://mlsanity.com/paper/2505.19590/) (2025)
- [Spurious Rewards: Rethinking Training Signals in RLVR](https://mlsanity.com/paper/2506.10947/) (2025)
- [The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning](https://mlsanity.com/paper/2505.15134/) (2025)
- [OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text](https://mlsanity.com/paper/2310.06786/) (2023)
- [REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2505.24760/) (2025)
- [OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling](https://mlsanity.com/paper/2506.20512/) (2025)
- [Benchmarking Benchmark Leakage in Large Language Models](https://mlsanity.com/paper/2404.18824/) (2024)
- [Maximizing Confidence Alone Improves Reasoning](https://mlsanity.com/paper/2505.22660/) (2025)
- [Can Large Reasoning Models Self-Train?](https://mlsanity.com/paper/2505.21444/) (2025)
- [MathPile: A Billion-Token-Scale Pretraining Corpus for Math](https://mlsanity.com/paper/2312.17120/) (2023)
- [Are Your LLMs Capable of Stable Reasoning?](https://mlsanity.com/paper/2412.13147/) (2024)
- [Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models](https://mlsanity.com/paper/2506.06395/) (2025)
- [No Free Lunch: Rethinking Internal Feedback for LLM Reasoning](https://mlsanity.com/paper/2506.17219/) (2025)
- [InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning](https://mlsanity.com/paper/2409.12568/) (2024)
- [One-shot Entropy Minimization](https://mlsanity.com/paper/2505.20282/) (2025)
- [The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason](https://mlsanity.com/paper/2505.22653/) (2025)

## Built on by (most cited first, 30 of 82)

- [FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction](https://mlsanity.com/paper/2508.11987/) (2025)
- [From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Ones](https://mlsanity.com/paper/2509.25123/) (2025)
- [Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle](https://mlsanity.com/paper/2509.16679/) (2025)
- [CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents](https://mlsanity.com/paper/2511.02734/) (2025)
- [Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead](https://mlsanity.com/paper/2510.01624/) (2025)
- [Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward](https://mlsanity.com/paper/2512.16912/) (2025)
- [ParallelSearch: Train your LLMs to Decompose Query and Search Sub-queries in Parallel with Reinforcement Learning](https://mlsanity.com/paper/2508.09303/) (2025)
- [Cognitive Foundations for Reasoning and Their Manifestation in LLMs](https://mlsanity.com/paper/2511.16660/) (2025)
- [FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization](https://mlsanity.com/paper/2603.19835/) (2026)
- [Revisiting Entropy in Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2511.05993/) (2025)
- [Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gain](https://mlsanity.com/paper/2603.02218/) (2026)
- [Evaluation Awareness Scales Predictably in Open-Weights Large Language Models](https://mlsanity.com/paper/2509.13333/) (2025)
- [Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs](https://mlsanity.com/paper/2601.11061/) (2026)
- [Learning to Reason in 13 Parameters](https://mlsanity.com/paper/2602.04118/) (2026)
- [Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2509.21882/) (2025)
- [Rethinking Reward Models for Multi-Domain Test-Time Scaling](https://mlsanity.com/paper/2510.00492/) (2025)
- [Benchmarking Multi-Step Legal Reasoning and Analyzing Chain-of-Thought Effects in Large Language Models](https://mlsanity.com/paper/2511.07979/) (2025)
- [OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning](https://mlsanity.com/paper/2511.23269/) (2025)
- [VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training](https://mlsanity.com/paper/2508.03058/) (2025)
- [On The Fragility of Benchmark Contamination Detection in Reasoning Models](https://mlsanity.com/paper/2510.02386/) (2025)
- [Beyond Pass@k: Breadth-Depth Metrics for Reasoning Boundaries](https://mlsanity.com/paper/2510.08325/) (2025)
- [BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Models](https://mlsanity.com/paper/2509.24210/) (2025)
- [R-Horizon: How Far Can Your Large Reasoning Model Really Go in Breadth and Depth?](https://mlsanity.com/paper/2510.08189/) (2025)
- [Tiny Model, Big Logic: Diversity-Driven Optimization Elicits Large-Model Reasoning Ability in VibeThinker-1.5B](https://mlsanity.com/paper/2511.06221/) (2025)
- [Better LLM Reasoning via Dual-Play](https://mlsanity.com/paper/2511.11881/) (2025)
- [Reinforced Efficient Reasoning via Semantically Diverse Exploration](https://mlsanity.com/paper/2601.05053/) (2026)
- [OpenSIR: Open-Ended Self-Improving Reasoner](https://mlsanity.com/paper/2511.00602/) (2025)
- [Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration](https://mlsanity.com/paper/2511.00794/) (2025)
- [Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2511.04800/) (2025)
- [BOAD: Discovering Hierarchical Software Engineering Agents via Bandit Optimization](https://mlsanity.com/paper/2512.23631/) (2025)

## Abstract (arXiv)

Reasoning in large language models has long been a central research focus, and recent studies employing reinforcement learning (RL) have introduced diverse methods that yield substantial performance gains with minimal or even no external supervision. Surprisingly, some studies even suggest that random or incorrect reward signals can enhance performance. However, these breakthroughs are predominantly observed for the mathematically strong Qwen2.5 series on benchmarks such as MATH-500, AMC, and AIME, and seldom transfer to models like Llama, which warrants a more in-depth investigation. In this work, our empirical analysis reveals that pre-training on massive web-scale corpora leaves Qwen2.5 susceptible to data contamination in widely used benchmarks. Consequently, conclusions derived from contaminated benchmarks on Qwen2.5 series may be unreliable. To obtain trustworthy evaluation results, we introduce a generator that creates fully clean arithmetic problems of arbitrary length and difficulty, dubbed RandomCalculation. Using this leakage-free dataset, we show that only accurate reward signals yield steady improvements that surpass the base model's performance boundary in mathematical reasoning, whereas random or incorrect rewards do not. Moreover, we conduct more fine-grained analyses to elucidate the factors underlying the different performance observed on the MATH-500 and RandomCalculation benchmarks. Consequently, we recommend that future studies evaluate models on uncontaminated benchmarks and, when feasible, test various model series to ensure trustworthy conclusions about RL and related methods.