# LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignment

arXiv: https://arxiv.org/abs/2506.11480 · HTML: https://mlsanity.com/paper/2506.11480/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 27 of its 38 references that are in the map.
- First posted on arXiv 13 Jun 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 27 of 27)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [LIMO: Less is More for Reasoning](https://mlsanity.com/paper/2502.03387/) (2025)
- [From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://mlsanity.com/paper/2308.12032/) (2023)
- [CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution](https://mlsanity.com/paper/2401.03065/) (2024)
- [Reinforcement Learning for Reasoning in Large Language Models with One Training Example](https://mlsanity.com/paper/2504.20571/) (2025)
- [Automatic Goal Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/1705.06366/) (2017)
- [The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset](https://mlsanity.com/paper/2303.03915/) (2023)
- [LIMR: Less is More for RL Scaling](https://mlsanity.com/paper/2502.11886/) (2025)
- [#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models](https://mlsanity.com/paper/2308.07074/) (2023)
- [Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning](https://mlsanity.com/paper/2504.03380/) (2025)
- [CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models](https://mlsanity.com/paper/2503.22342/) (2025)
- [Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models](https://mlsanity.com/paper/2405.20541/) (2024)
- [What Makes a Reward Model a Good Teacher? An Optimization Perspective](https://mlsanity.com/paper/2503.15477/) (2025)
- [M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy](https://mlsanity.com/paper/2312.15927/) (2023)
- [Optimizing Data Usage via Differentiable Rewards](https://mlsanity.com/paper/1911.10088/) (2019)
- [Large-Scale Data Selection for Instruction Tuning](https://mlsanity.com/paper/2503.01807/) (2025)
- [First is Better Than Last for Language Data Influence](https://mlsanity.com/paper/2202.11844/) (2022)
- [DANCE: Dual-View Distribution Alignment for Dataset Condensation](https://mlsanity.com/paper/2406.01063/) (2024)
- [Neural Networks for Learnable and Scalable Influence Estimation of Instruction Fine-Tuning Data](https://mlsanity.com/paper/2502.09969/) (2025)
- [A Two-Stage Data Selection Framework for Data-Efficient Model Training on Edge Devices](https://mlsanity.com/paper/2505.16563/) (2025)

## Built on by (most cited first, 9 of 9)

- [SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoning](https://mlsanity.com/paper/2601.04809/) (2026)
- [AgenticMath: Enhancing LLM Reasoning via Agentic-based Math Data Generation](https://mlsanity.com/paper/2510.19361/) (2025)
- [Self-Improvement of Large Language Models: A Technical Overview and Future Outlook](https://mlsanity.com/paper/2603.25681/) (2026)
- [Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2608.16647/) (2026)
- [Guiding LLM Post-training Data Engineering with Model Internals from Sparse Autoencoders](https://mlsanity.com/paper/2605.27354/) (2026)
- [Denoising Implicit Feedback for Cold-start Recommendation](https://mlsanity.com/paper/2606.19658/) (2026)
- [CompassPlay: Rewarding the Proposer for Where It Moves the Solver](https://mlsanity.com/paper/2609.32228/) (2026)
- [Not All Rollouts Are Worth Learning: On Trajectory Valuation for Post-Training Reinforcement Learning](https://mlsanity.com/paper/2609.35072/) (2026)
- [SelectTTRL: Representation-Guided Data Selection for Efficient Test-Time Reinforcement Learning](https://mlsanity.com/paper/2610.16104/) (2026)

## Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has become a key technique for enhancing LLMs' reasoning abilities, yet its data inefficiency remains a major bottleneck. To address this critical yet challenging issue, we present a novel gradient-alignment-based method, named LearnAlign, which intelligently selects the learnable and representative training reasoning data for RLVR post-training. To overcome the well-known response-length bias in gradient norms, we introduce the data learnability based on the success rate, which indicates the learning potential of each data point. Experiments across five reasoning benchmarks show that our method significantly reduces training data requirements while achieving minor performance degradation or even improving performance compared to full-data training. Specifically, it reduces data requirements by up to 1,000 data points with better performance (77.5%) than that on the full dataset on the GSM8K benchmark (77.0%). Furthermore, its efficiency is demonstrated on both mathematical and code benchmarks by using much less data from the DAPO-MATH-17K dataset.