# RL-Guided Data Selection for Language Model Finetuning

arXiv: https://arxiv.org/abs/2509.25850 · HTML: https://mlsanity.com/paper/2509.25850/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 33 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Train faster, generalize better: Stability of stochastic gradient descent](https://mlsanity.com/paper/1509.01240/) (2015)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining](https://mlsanity.com/paper/2305.10429/) (2023)
- [Data Selection for Language Models via Importance Resampling](https://mlsanity.com/paper/2302.03169/) (2023)
- [SemDeDup: Data-efficient learning at web-scale through semantic deduplication](https://mlsanity.com/paper/2303.09540/) (2023)
- [A Survey on Data Selection for Language Models](https://mlsanity.com/paper/2402.16827/) (2024)
- [MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases](https://mlsanity.com/paper/2402.14905/) (2024)
- [A Closer Look at Invalid Action Masking in Policy Gradient Algorithms](https://mlsanity.com/paper/2006.14171/) (2020)
- [D4: Improving LLM Pretraining via Document De-Duplication and Diversification](https://mlsanity.com/paper/2308.12284/) (2023)
- [Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training](https://mlsanity.com/paper/2504.13161/) (2025)
- [Data Selection via Optimal Control for Language Models](https://mlsanity.com/paper/2410.07064/) (2024)
- [Towards a statistical theory of data selection under weak supervision](https://mlsanity.com/paper/2309.14563/) (2023)
- [FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain](https://mlsanity.com/paper/2505.14826/) (2025)
- [Efficient Data Selection at Scale via Influence Distillation](https://mlsanity.com/paper/2505.19051/) (2025)
- [MetaHate: A Dataset for Unifying Efforts on Hate Speech Detection](https://mlsanity.com/paper/2401.06526/) (2024)
- [Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models](https://mlsanity.com/paper/2406.14115/) (2024)
- [Progressive Data Dropout: An Embarrassingly Simple Approach to Faster Training](https://mlsanity.com/paper/2505.22342/) (2025)
- [Learning What Matters: Prioritized Concept Learning via Relative Error-driven Sample Selection](https://mlsanity.com/paper/2506.01085/) (2025)

## Built on by (most cited first, 2 of 2)

- [Learning What to Fail On: Failure-Mode Contextual Bandits for Adversarial Data Curation](https://mlsanity.com/paper/2608.18681/) (2026)
- [Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling](https://mlsanity.com/paper/2609.32484/) (2026)

## Abstract (arXiv)

Data selection for finetuning Large Language Models (LLMs) can be framed as a budget-constrained optimization problem: maximizing a model's downstream performance under a strict training data budget. Solving this problem is generally intractable, and existing approximate approaches are pretraining-oriented and transfer poorly to the fine-tuning setting. We reformulate this problem as a tractable Markov Decision Process (MDP) and train agents using various Reinforcement Learning (RL) methods to learn optimal data selection policies, guided by an efficient, proxy-model-based reward signal. Across four datasets, training on a 5% subset selected by our approach matches or outperforms fine-tuning on the full dataset by up to 10.8 accuracy points, while cutting wall-clock training time by up to 2 ×, highlighting the promise of RL-guided data selection.