mlsanityData to 7 Oct 2026

The Unlearnability Phenomenon in RLVR for Language Models

arXiv 2605.16787 ↗First posted 16 May 2026cs.LG cs.CLcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Reward (RLVR) has proven effective in improving Large Language Model's (LLM) reasoning ability. However, the learning dynamics of RLVR remain underexplored. In this paper, we reveal a counterintuitive phenomenon: among hard examples that the model initially struggles with, a substantial subset remains unlearnable even when correct rollouts are present. To understand the phenomenon, we first demonstrate that existing optimization and sampling techniques fail to resolve unlearnability. With cross-example gradient analysis, we show that unlearnable examples have fundamental representation issue, characterized by low gradient similarity with the rest of the examples and ungeneralizable reasoning patterns. We further show that representation flaws are difficult to mitigate in RL, as data augmentation does not improve gradient similarity. Our study provides the first systematic characterization of unlearnable data in RLVR training and reveals fundamental limitations in current RL approaches for reasoning tasks. Code and data are available at urlhttps://github.com/yulinchen99/unlearnability-rlvr.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 30 references that are in the map. First posted on arXiv 16 May 2026.

Builds on

22 of 30 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models8 Dec
  2. 2025How Does RL Post-training Induce Skill Composition? A Case Study on Countdown1 Dec
  3. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models8 Nov
  4. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models2 Oct
  5. 2025Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse30 Sep
  6. 2025RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning10 Jul
  7. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  8. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay5 Jun
  9. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  10. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  11. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  12. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learning7 Apr
3 more
  1. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  2. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar

Other references · 7

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 268
  4. 2025Reasoning Models Know When They're Right: Probing Hidden States for Self-Verificationcited by 154
  5. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 94
  6. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 92
  7. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026DRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer Trainingcited by 2
  2. 2026Unlearnable, or Unmeasured? On the Reliability of Difficulty Labels in RLVRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation
  2. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation
  3. 2026Linear Dynamics in the RLVR Training of Large Language Models
  4. 2025The Path Not Taken: RLVR Provably Learns Off the Principals
  5. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  6. 2025Limits of Generalization in RLVR: Two Case Studies in Mathematical Reasoning
  7. 2026Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes
  8. 2026Understanding Diversity Collapse in RLVR via the Lens of Overtraining