mlsanityData to 14 Oct 2026

LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignment

arXiv 2506.11480 ↗First posted 13 Jun 2025cs.LG cs.AIcited by 9

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has become a key technique for enhancing LLMs' reasoning abilities, yet its data inefficiency remains a major bottleneck. To address this critical yet challenging issue, we present a novel gradient-alignment-based method, named LearnAlign, which intelligently selects the learnable and representative training reasoning data for RLVR post-training. To overcome the well-known response-length bias in gradient norms, we introduce the data learnability based on the success rate, which indicates the learning potential of each data point. Experiments across five reasoning benchmarks show that our method significantly reduces training data requirements while achieving minor performance degradation or even improving performance compared to full-data training. Specifically, it reduces data requirements by up to 1,000 data points with better performance (77.5%) than that on the full dataset on the GSM8K benchmark (77.0%). Furthermore, its efficiency is demonstrated on both mathematical and code benchmarks by using much less data from the DAPO-MATH-17K dataset.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 27 of its 38 references that are in the map. First posted on arXiv 13 Jun 2025.

Builds on

27 of 38 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  2. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning4 Apr
  3. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  5. 2025LIMR: Less is More for RL Scaling17 Feb
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 21

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2023LIMA: Less Is More for Alignmentcited by 1,100
  5. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  6. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  7. 2025LIMO: Less is More for Reasoningcited by 498
  8. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  9. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  10. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  11. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  12. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 109
9 more
  1. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 84
  2. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 73
  3. 2023M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancycited by 49
  4. 2019Optimizing Data Usage via Differentiable Rewardscited by 48
  5. 2025Large-Scale Data Selection for Instruction Tuningcited by 35
  6. 2022First is Better Than Last for Language Data Influencecited by 34
  7. 2024DANCE: Dual-View Distribution Alignment for Dataset Condensationcited by 15
  8. 2025Neural Networks for Learnable and Scalable Influence Estimation of Instruction Fine-Tuning Datacited by 2
  9. 2025A Two-Stage Data Selection Framework for Data-Efficient Model Training on Edge Devicescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 8.

9 papers.

  1. 2026SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoningcited by 8
  2. 2025AgenticMath: Enhancing LLM Reasoning via Agentic-based Math Data Generationcited by 7
  3. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  4. 2026Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Modelscited by 4
  5. 2026Guiding LLM Post-training Data Engineering with Model Internals from Sparse Autoencoderscited by 2
  6. 2026Denoising Implicit Feedback for Cold-start Recommendationcited by 0
  7. 2026CompassPlay: Rewarding the Proposer for Where It Moves the Solvercited by 0
  8. 2026Not All Rollouts Are Worth Learning: On Trajectory Valuation for Post-Training Reinforcement Learningcited by 0
  9. 2026SelectTTRL: Representation-Guided Data Selection for Efficient Test-Time Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning
  2. 2026Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards
  3. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
  4. 2025TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning
  5. 2026References Improve LLM Alignment in Non-Verifiable Domains
  6. 2026SUPERNOVA: Eliciting General Reasoning in LLMs with Reinforcement Learning on Natural Instructions
  7. 2025Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward
  8. 2026f-GRPO and Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignment