# Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs

arXiv: https://arxiv.org/abs/2601.08763 · HTML: https://mlsanity.com/paper/2601.08763/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 13 Jan 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 9 of 9)

- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [Reasoning with Exploration: An Entropy Perspective](https://mlsanity.com/paper/2506.14758/) (2025)
- [SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization](https://mlsanity.com/paper/2505.12346/) (2025)
- [Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems](https://mlsanity.com/paper/2505.15201/) (2025)
- [Jointly Reinforcing Diversity and Quality in Language Model Generations](https://mlsanity.com/paper/2509.02534/) (2025)
- [Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward](https://mlsanity.com/paper/2510.03222/) (2025)

## Built on by (most cited first, 12 of 12)

- [EvoClinician: A Self-Evolving Agent for Multi-Turn Medical Diagnosis via Test-Time Evolutionary Learning](https://mlsanity.com/paper/2601.22964/) (2026)
- [Poly-EPO: Training Exploratory Reasoning Models](https://mlsanity.com/paper/2604.17654/) (2026)
- [Reasoning over mathematical objects: on-policy reward modeling and test time aggregation](https://mlsanity.com/paper/2603.18886/) (2026)
- [BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?](https://mlsanity.com/paper/2608.02867/) (2026)
- [SuS: Strategy-aware Surprise for Intrinsic Exploration](https://mlsanity.com/paper/2601.10349/) (2026)
- [The Past Is Not Past: Memory-Enhanced Dynamic Reward Shaping](https://mlsanity.com/paper/2604.11297/) (2026)
- [The Model Knows Another Way: Strategy Switching for Effective RLVR Exploration](https://mlsanity.com/paper/2609.33085/) (2026)
- [Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards](https://mlsanity.com/paper/2605.14539/) (2026)
- [When RL Suppresses Its Own Vocabulary: Recovering Reasoning Diversity in Puzzle-to-Math Transfer](https://mlsanity.com/paper/2605.29190/) (2026)
- [Expert-Space Exploration in MoE Reinforcement Learning](https://mlsanity.com/paper/2609.13058/) (2026)
- [Selecting Diverse SFT Traces Improves Post-RL Generalization](https://mlsanity.com/paper/2609.33780/) (2026)
- [LLMs as Adaptive Meta-Solvers: Strategy-Diverse RL for Industrial-Scale Optimization](https://mlsanity.com/paper/2609.34427/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) has become a central paradigm for post-training large language models (LLMs), particularly for complex reasoning tasks, yet it often suffers from exploration collapse: policies prematurely concentrate on a small set of dominant reasoning patterns, improving pass@1 while limiting rollout-level diversity and gains in pass@k. We argue that this failure stems from regularizing local token behavior rather than diversity over sets of solutions. To address this, we propose Uniqueness-Aware Reinforcement Learning, a rollout-level objective that explicitly rewards correct solutions that exhibit rare high-level strategies. Our method uses an LLM-based judge to cluster rollouts for the same problem according to their high-level solution strategies, ignoring superficial variations, and reweights policy advantages inversely with cluster size. As a result, correct but novel strategies receive higher rewards than redundant ones. Across mathematics, physics, and medical reasoning benchmarks, our approach consistently improves pass@k across large sampling budgets and increases the area under the pass@k curve (AUC@K) without sacrificing pass@1, while sustaining exploration and uncovering more diverse solution strategies at scale.