mlsanityData to 14 Oct 2026

RL-Guided Data Selection for Language Model Finetuning

arXiv 2509.25850 ↗First posted 30 Sep 2025cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Data selection for finetuning Large Language Models (LLMs) can be framed as a budget-constrained optimization problem: maximizing a model's downstream performance under a strict training data budget. Solving this problem is generally intractable, and existing approximate approaches are pretraining-oriented and transfer poorly to the fine-tuning setting. We reformulate this problem as a tractable Markov Decision Process (MDP) and train agents using various Reinforcement Learning (RL) methods to learn optimal data selection policies, guided by an efficient, proxy-model-based reward signal. Across four datasets, training on a 5% subset selected by our approach matches or outperforms fine-tuning on the full dataset by up to 10.8 accuracy points, while cutting wall-clock training time by up to 2 ×, highlighting the promise of RL-guided data selection.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 26 of its 33 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

26 of 33 references.

References · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  6. 2018Exploration by Random Network Distillationcited by 993
  7. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  8. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  9. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  10. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  11. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  12. 2023Data Selection for Language Models via Importance Resamplingcited by 329
14 more
  1. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  2. 2024A Survey on Data Selection for Language Modelscited by 236
  3. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  4. 2020A Closer Look at Invalid Action Masking in Policy Gradient Algorithmscited by 171
  5. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 150
  6. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 66
  7. 2024Data Selection via Optimal Control for Language Modelscited by 38
  8. 2023Towards a statistical theory of data selection under weak supervisioncited by 27
  9. 2025FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gaincited by 15
  10. 2025Efficient Data Selection at Scale via Influence Distillationcited by 15
  11. 2024MetaHate: A Dataset for Unifying Efforts on Hate Speech Detectioncited by 14
  12. 2024Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Modelscited by 12
  13. 2025Progressive Data Dropout: An Embarrassingly Simple Approach to Faster Trainingcited by 4
  14. 2025Learning What Matters: Prioritized Concept Learning via Relative Error-driven Sample Selectioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Learning What to Fail On: Failure-Mode Contextual Bandits for Adversarial Data Curationcited by 0
  2. 2026Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Samplingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions
  2. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay
  3. 2025RL-finetuning LLMs from on- and off-policy data with a single algorithm
  4. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  5. 2026Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models
  6. 2024Mixture-of-Skills: Learning to Optimize Data Usage for Fine-Tuning Large Language Models
  7. 2025LEAD: Iterative Data Selection for Efficient LLM Instruction Tuning
  8. 2025Reinforcement Learning Fine-Tunes a Sparse Subnetwork in Large Language Models