# Efficient RLVR Training via Weighted Mutual Information Data Selection

arXiv: https://arxiv.org/abs/2603.01907 · HTML: https://mlsanity.com/paper/2603.01907/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 2 Mar 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 27 of 27)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [From System 1 to System 2: A Survey of Reasoning Large Language Models](https://mlsanity.com/paper/2502.17419/) (2025)
- [LIMR: Less is More for RL Scaling](https://mlsanity.com/paper/2502.11886/) (2025)
- [General-Reasoner: Advancing LLM Reasoning Across All Domains](https://mlsanity.com/paper/2505.14652/) (2025)
- [Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning](https://mlsanity.com/paper/2504.03380/) (2025)
- [Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning](https://mlsanity.com/paper/2506.06632/) (2025)
- [Estimating Epistemic and Aleatoric Uncertainty with a Single Model](https://mlsanity.com/paper/2402.03478/) (2024)
- [Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR](https://mlsanity.com/paper/2508.14029/) (2025)
- [Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?](https://mlsanity.com/paper/2507.04632/) (2025)
- [Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking](https://mlsanity.com/paper/2403.03185/) (2024)
- [TreeRPO: Tree Relative Policy Optimization](https://mlsanity.com/paper/2506.05183/) (2025)
- [Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration](https://mlsanity.com/paper/2508.13755/) (2025)
- [BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning](https://mlsanity.com/paper/2510.26374/) (2025)
- [HyperINF: Unleashing the HyperPower of the Schulz's Method for Data Influence Estimation](https://mlsanity.com/paper/2410.05090/) (2024)
- [How Many Code and Test Cases Are Enough? Evaluating Test Cases Generation from a Binary-Matrix Perspective](https://mlsanity.com/paper/2510.08720/) (2025)

## Built on by (most cited first, 6 of 6)

- [Low-rank Optimization Trajectories Modeling for LLM RLVR Acceleration](https://mlsanity.com/paper/2604.11446/) (2026)
- [Faster Synchronous On-Policy RL via Straggler-Aware Group Sizing](https://mlsanity.com/paper/2606.02218/) (2026)
- [RLVR Datasets and Where to Find Them: Tracing Data Lineage for Better Training Data](https://mlsanity.com/paper/2605.26971/) (2026)
- [MaPP: A Unified Marginalized Posterior-Predictive Framework for Data-Efficient RLVR](https://mlsanity.com/paper/2609.34990/) (2026)
- [RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillation](https://mlsanity.com/paper/2610.00979/) (2026)
- [A Step Harder, Not a Step Too Far: The Benefits and Limits of Curriculum Difficulty Progression in GRPO](https://mlsanity.com/paper/2610.13669/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) plays a central role in improving the reasoning and alignment of large language models, yet its efficiency critically depends on how training data are selected. Existing online selection strategies predominantly rely on difficulty-based heuristics, favouring datapoints with intermediate success rates, implicitly equating difficulty with informativeness and neglecting epistemic uncertainty arising from limited evidence. We introduce InSight, an INformation-guided data SamplInG metHod for RL Training, grounded in a weighted mutual information objective. By modeling data outcomes with Bayesian latent success rates, we show that expected uncertainty reduction decomposes into complementary difficulty- and evidence-dependent components, revealing a fundamental limitation of difficulty-only selection. Leveraging this observation, InSight constructs a stable acquisition score based on the mean belief of datapoints' success rather than noisy sampled outcomes, and naturally extends to multi-rollout settings common in reinforcement learning with verifiable rewards (RLVR). Extensive experiments demonstrate that InSight consistently achieves state-of-the-art performance and improves training efficiency, including a +1.41 average gain on Planning & Mathmatics benchmarks, +1.01 improvement on general reasoning, and up to ~2.2x acceleration, with negligible additional computational overhead.