mlsanityData to 7 Oct 2026

DavIR: Data Selection via Implicit Reward for Large Language Models

arXiv 2310.13008 ↗First posted 16 Oct 2023cs.LG cs.AI cs.CLcited by 6

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We introduce DavIR, a model-based data selection method for post-training Large Language Models. DavIR generalizes Reducible Holdout Loss to core-set selection problem of causal language modeling, and quantifies the learnability of a given datum with respect to a pre-trained LLM based on relative reduction in loss during fine-tuning, a metric we show to be closely related to the implicit reward model described in Direct Preference Optimization (DPO). We show that 6% of Alpaca dataset selected with DavIR can steer both the LLaMA and Gemma model family to produce superior performance compared to the same models trained on the full 52K dataset. We also show that Alpaca dataset compressed with DavIR can be combined with GSM8K dataset to effectively balance open-domain freeform QA and mathematical reasoning capabilities. Finally, we apply the DavIR objective to DPO and develop a normalized DavIR-DPO objective which improves alignment performance of Zephyr-7B-SFT model by 8% (relative) on AlpacaEval, compared against training on vanilla DPO objective.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 53 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

28 of 53 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  2. 2023LIMA: Less Is More for Alignment18 May
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  10. 2018Neural Network Acceptability Judgmentscited by 952
  11. 2023BloombergGPT: A Large Language Model for Financecited by 923
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
11 more
  1. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  2. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  3. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  4. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  5. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  6. 2019Distributionally Robust Language Modelingcited by 142
  7. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 132
  8. 2023Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Casescited by 85
  9. 2021RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learningcited by 72
  10. 2023Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuningcited by 56
  11. 2019Semantic Redundancies in Image-Classification Datasets: The 10% You Don't Needcited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 1; 2026: 2.

6 papers.

  1. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  2. 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 43
  3. 2023Data Management For Training Large Language Models: A Surveycited by 17
  4. 2025Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-Tuning via Closed-Loop Learningcited by 7
  5. 2026One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuningcited by 0
  6. 2026Video Understanding Reward Modeling: A Robust Benchmark and Performant Reward Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimization
  2. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  3. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
  4. 2024Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models
  5. 2025Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
  6. 2024SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins
  7. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignment
  8. 2025When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets