mlsanityData to 14 Oct 2026

Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs

arXiv 2601.11061 ↗First posted 16 Jan 2026cs.LG cs.CLcited by 14

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) is highly effective for enhancing LLM reasoning, yet recent evidence shows models like Qwen 2.5 achieve significant gains even with spurious or incorrect rewards. We investigate this phenomenon and identify a "Perplexity Paradox": spurious RLVR triggers a divergence where answer-token perplexity drops while prompt-side coherence degrades, suggesting the model is bypassing reasoning in favor of memorization. Using Path Patching, Logit Lens, JSD analysis, and Neural Differential Equations, we uncover a hidden Anchor-Adapter circuit that facilitates this shortcut. We localize a Functional Anchor in the middle layers (L18-20) that triggers the retrieval of memorized solutions, followed by Structural Adapters in later layers (L21+) that transform representations to accommodate the shortcut signal. Finally, we demonstrate that scaling specific MLP keys within this circuit allows for bidirectional causal steering-artificially amplifying or suppressing contamination-driven performance. Our results provide a mechanistic roadmap for identifying and mitigating data contamination in RLVR-tuned models. Code is available at https://github.com/idwts/How-RLVR-Activates-Memorization-Shortcuts.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 31 references that are in the map. First posted on arXiv 16 Jan 2026.

Builds on

26 of 31 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward18 Dec
  2. 2025Limits of Generalization in RLVR: Two Case Studies in Mathematical Reasoning30 Oct
  3. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination14 Jul
  4. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  5. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  6. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  7. 2023Let's Verify Step by Step31 May

Other references · 19

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2018Neural Ordinary Differential Equationscited by 3,779
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  8. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  9. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 113
  10. 2024ContextCite: Attributing Model Generation to Contextcited by 98
  11. 2024Are Your LLMs Capable of Stable Reasoning?cited by 65
  12. 2024Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Datacited by 36
7 more
  1. 2025The Reasoning-Memorization Interplay in Language Models Is Mediated by a Single Directioncited by 21
  2. 2024Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisitioncited by 16
  3. 2025Attributing Response to Context: A Jensen-Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generationcited by 12
  4. 2025Unveiling Over-Memorization in Finetuning LLMs for Reasoning Taskscited by 11
  5. 2025HD-NDEs: Neural Differential Equations for Hallucination Detection in LLMscited by 6
  6. 2025Mirage or Method? How Model-Task Alignment Induces Divergent RL Conclusionscited by 6
  7. 2024Detecting Memorization in Large Language Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 14.

14 papers.

Sort:

By area:

  1. 2026CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVRcited by 5
  2. 2026Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Trainingcited by 4
  3. 2026Survive or Collapse: The Asymmetric Roles of Data Gating and Reward Grounding in Self-Play RLcited by 4
  4. 2026VeriGate: Verifier-Gated Step-Level Supervision for GRPOcited by 4
  5. 2026When Does a Video-Language Model Stop Watching? Reward Strength Controls the Formation and Reversal of Visual Shortcuts in Multimodal RLVRcited by 2
  6. 2026Tracing the Latent Threads: A Mechanistic Study of How LLMs Represent and Operationalize Race and Ethnicity Cuescited by 1
  7. 2026To Reason or to Fabricate: Reasoning Without Shortcuts via Hint-Anchored Pairwise Aggregationcited by 0
  8. 2026Predictable GRPO: A Closed-Form Model of Training Dynamicscited by 0
  9. 2026Bridging Physical Reasoning and Task Generalization via Visual Action Outcome Reasoning Alignmentcited by 0
  10. 2026Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Spacecited by 0
  11. 2026Patterns of Priming in Production: Lexical, Semantic and Structural Alignment in Language Model Generationcited by 0
  12. 2026When Does Correction Become Repair? Mechanistic Auditing of Internal Interventions in Tool-Using LLMscited by 0
2 more
  1. 2026MetaSteer: Context-Conditioned, nonlinear Steering via Attention-Projection Adaptationcited by 0
  2. 2026SFT Leaves a Reproducible Signature in Activation Geometry: Calibrating Post-Training Interpretability Against Run-to-Run Noisecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  2. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  3. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation
  4. 2026Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
  5. 2025RLPR: Extrapolating RLVR to General Domains without Verifiers
  6. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
  7. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
  8. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origin