mlsanityData to 7 Oct 2026

R.I.P.: Better Models by Survival of the Fittest Prompts

arXiv 2501.18578 ↗First posted 30 Jan 2025cs.CL cs.AI cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Training data quality is one of the most important drivers of final model quality. In this work, we introduce a method for evaluating data integrity based on the assumption that low-quality input prompts result in high variance and low quality responses. This is achieved by measuring the rejected response quality and the reward gap between the chosen and rejected preference pair. Our method, Rejecting Instruction Preferences (RIP) can be used to filter prompts from existing training sets, or to make high quality synthetic datasets, yielding large performance gains across various benchmarks compared to unfiltered data. Using Llama 3.1-8B-Instruct, RIP improves AlpacaEval2 LC Win Rate by 9.4%, Arena-Hard by 8.7%, and WildBench by 9.9%. Using Llama 3.3-70B-Instruct, RIP improves Arena-Hard from 67.5 to 82.9, which is from 18th place to 6th overall in the leaderboard.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 40 of its 46 references that are in the map. First posted on arXiv 30 Jan 2025.

Builds on

40 of 46 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
28 more
  1. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  4. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  5. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  6. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  7. 2024Self-Rewarding Language Modelscited by 654
  8. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  9. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  10. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  11. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  12. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  13. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  14. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  15. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  16. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  17. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  18. 2023Self-Alignment with Instruction Backtranslationcited by 149
  19. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 134
  20. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 132
  21. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 108
  22. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  23. 2024β-DPO: Direct Preference Optimization with Dynamic βcited by 89
  24. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85
  25. 2024Self-Taught Evaluatorscited by 70
  26. 2023Instruction Mining: Instruction Data Selection for Tuning Large Language Modelscited by 57
  27. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Modelscited by 48
  28. 2024Following Length Constraints in Instructionscited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 3.

13 papers.

Sort:

By area:

  1. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
  2. 2025CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning taskscited by 30
  3. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  4. 2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Insteadcited by 24
  5. 2025The Era of Real-World Human Interaction: RL from User Conversationscited by 13
  6. 2025AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Datasetcited by 8
  7. 2025RLTHF: Targeted Human Feedback for LLM Alignmentcited by 7
  8. 2025Less is More: Improving LLM Alignment via Preference Data Selectioncited by 4
  9. 2025Tailored Primitive Initialization is the Secret Key to Reinforcement Learningcited by 4
  10. 2026A Statistical Framework for Alignment with Biased AI Feedbackcited by 4
  11. 2025Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learningcited by 1
  12. 2026CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Productioncited by 1
1 more
  1. 2026Expectation Alignment of Language Models for Real-World User Expectationscited by 0