# ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuning

arXiv: https://arxiv.org/abs/2412.00631 · HTML: https://mlsanity.com/paper/2412.00631/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 62 references that are in the map.
- First posted on arXiv 1 Dec 2024.

## Builds on (most cited first, 30 of 35)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning](https://mlsanity.com/paper/2312.15685/) (2023)
- [Data Selection for Language Models via Importance Resampling](https://mlsanity.com/paper/2302.03169/) (2023)
- [From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://mlsanity.com/paper/2308.12032/) (2023)
- [TRAK: Attributing Model Behavior at Scale](https://mlsanity.com/paper/2303.14186/) (2023)
- [Pretraining Language Models with Human Preferences](https://mlsanity.com/paper/2302.08582/) (2023)
- [Fine-tuning Language Models for Factuality](https://mlsanity.com/paper/2311.08401/) (2023)
- [Post-hoc Interpretability for Neural NLP: A Survey](https://mlsanity.com/paper/2108.04840/) (2021)
- [Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss](https://mlsanity.com/paper/2312.16682/) (2023)
- [MoDS: Model-oriented Data Selection for Instruction Tuning](https://mlsanity.com/paper/2311.15653/) (2023)
- [Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers](https://mlsanity.com/paper/2109.10686/) (2021)
- [LiPO: Listwise Preference Optimization through Learning-to-Rank](https://mlsanity.com/paper/2402.01878/) (2024)
- [A Survey on Data Selection for LLM Instruction Tuning](https://mlsanity.com/paper/2402.05123/) (2024)
- [MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets](https://mlsanity.com/paper/2308.12067/) (2023)
- [Shapley values for feature selection: The good, the bad, and the axioms](https://mlsanity.com/paper/2102.10936/) (2021)
- [Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Models](https://mlsanity.com/paper/2402.10430/) (2024)

## Built on by (most cited first, 10 of 10)

- [CrowdSelect: Synthetic Instruction Data Selection with Multi-LLM Wisdom](https://mlsanity.com/paper/2503.01836/) (2025)
- [Teaching According to Students' Aptitude: Personalized Mathematics Tutoring via Persona-, Memory-, and Forgetting-Aware LLMs](https://mlsanity.com/paper/2511.15163/) (2025)
- [Active Domain Knowledge Acquisition with 100-Dollar Budget: Enhancing LLMs via Cost-Efficient, Expert-Involved Interaction in Sensitive Domains](https://mlsanity.com/paper/2508.17202/) (2025)
- ["Excuse me, may I say something..." CoLabScience, A Proactive AI Assistant for Biomedical Discovery and LLM-Expert Collaborations](https://mlsanity.com/paper/2604.15588/) (2026)
- [CODEBLOCK: Learning to Supervise Code at the Right Granularity](https://mlsanity.com/paper/2606.18286/) (2026)
- [CLaDMoP: Learning Transferrable Models from Successful Clinical Trials via LLMs](https://mlsanity.com/paper/2505.18527/) (2025)
- [CLID-MU: Cross-Layer Information Divergence Based Meta Update Strategy for Learning with Noisy Labels](https://mlsanity.com/paper/2507.11807/) (2025)
- [SUPERNOVA: Eliciting General Reasoning in LLMs with Reinforcement Learning on Natural Instructions](https://mlsanity.com/paper/2604.08477/) (2026)
- [Let the Target Select for Itself: Data Selection via Target-Aligned Paths](https://mlsanity.com/paper/2605.09404/) (2026)
- [SemiAdapt-Instruct: Extensible Instruction Tuning via Latent Domain-Specialised Adapters](https://mlsanity.com/paper/2608.05161/) (2026)

## Abstract (arXiv)

Instruction tuning has underscored the significant potential of large language models (LLMs) in producing more human controllable and effective outputs in various domains. In this work, we focus on the data selection problem for task-specific instruction tuning of LLMs. Prevailing methods primarily rely on the crafted similarity metrics to select training data that aligns with the test data distribution. The goal is to minimize instruction tuning loss on the test data, ultimately improving performance on the target task. However, it has been widely observed that instruction tuning loss (i.e., cross-entropy loss for next token prediction) in LLMs often fails to exhibit a monotonic relationship with actual task performance. This misalignment undermines the effectiveness of current data selection methods for task-specific instruction tuning. To address this issue, we introduce ROSE, a novel Reward-Oriented inStruction data sElection method which leverages pairwise preference loss as a reward signal to optimize data selection for task-specific instruction tuning. Specifically, ROSE adapts an influence formulation to approximate the influence of training data points relative to a few-shot preference validation set to select the most task-related training data points. Experimental results show that by selecting just 5% of the training data using ROSE, our approach can achieve competitive results compared to fine-tuning with the full training dataset, and it surpasses other state-of-the-art data selection methods for task-specific instruction tuning. Our qualitative analysis further confirms the robust generalizability of our method across multiple benchmark datasets and diverse model architectures.