mlsanityData to 7 Oct 2026

Learn More with Less: Uncertainty Consistency Guided Query Selection for RLVR

arXiv 2601.22595 ↗First posted 30 Jan 2026cs.AIcited by 1

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large Language Models (LLMs) have recently improved mathematical reasoning through Reinforcement Learning with Verifiable Reward (RLVR). However, existing RLVR algorithms require large query budgets, making annotation costly. We investigate whether fewer but more informative queries can yield similar or superior performance, introducing active learning (AL) into RLVR. We identify that classic AL sampling strategies fail to outperform random selection in this setting, due to ignoring objective uncertainty when only selecting by subjective uncertainty. This work proposes an uncertainty consistency metric to evaluate how well subjective uncertainty aligns with objective uncertainty. In the offline setting, this alignment is measured using the Point-Biserial Correlation Coefficient (PBC). For online training, because of limited sampling and dynamically shifting output distributions, PBC estimation is difficult. Therefore, we introduce a new online variant, computed from normalized advantage and subjective uncertainty. Theoretically, we prove that the online variant is strictly negatively correlated with offline PBC and supports better sample selection. Experiments show our method consistently outperforms random and classic AL baselines, achieving full-dataset performance while training on only 30% of the data, effectively reducing the cost of RLVR for reasoning tasks.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 35 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

24 of 35 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  3. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 19

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  8. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  9. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  10. 2020Contextual Diversity for Active Learningcited by 139
  11. 2021Batch Active Learning at Scalecited by 127
  12. 2024How to Train Data-Efficient LLMscited by 91
7 more
  1. 2021Gone Fishing: Neural Active Learning with Fisher Embeddingscited by 88
  2. 2017Deep Active Learning over the Long Tailcited by 82
  3. 2024Active Preference Learning for Large Language Modelscited by 55
  4. 2024ActiveLLM: Large Language Model-based Active Learning for Textual Few-Shot Scenarioscited by 17
  5. 2024Causal-Guided Active Learning for Debiasing Large Language Modelscited by 15
  6. 2024Cost-Effective Proxy Reward Model Construction with On-Policy and Active Learningcited by 5
  7. 2023Interactive Multi-fidelity Learning for Cost-effective Adaptation of Language Model with Sparse Human Supervisioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026IRDS: Interpretable RLVR Data Selection via Verifier-Coupled Sparse Autoencoder Coveragecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient RLVR Training via Weighted Mutual Information Data Selection
  2. 2026When Self-Belief Misleads: Active Label Acquisition for Reinforcement Learning with Verifiable Rewards
  3. 2025Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning
  4. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  5. 2026Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation
  6. 2025Generalization of RLVR Using Causal Reasoning as a Testbed
  7. 2025Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Models
  8. 2026Linear Dynamics in the RLVR Training of Large Language Models