mlsanityData to 14 Oct 2026

RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment

arXiv 2304.06767 ↗First posted 13 Apr 2023cs.LG cs.AI cs.CLcited by 686

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Generative foundation models are susceptible to implicit biases that can arise from extensive unsupervised training data. Such biases can produce suboptimal samples, skewed outcomes, and unfairness, with potentially serious consequences. Consequently, aligning these models with human ethics and preferences is an essential step toward ensuring their responsible and effective deployment in real-world applications. Prior research has primarily employed Reinforcement Learning from Human Feedback (RLHF) to address this problem, where generative models are fine-tuned with RL algorithms guided by a human-feedback-informed reward model. However, the inefficiencies and instabilities associated with RL algorithms frequently present substantial obstacles to the successful alignment, necessitating the development of a more robust and streamlined approach. To this end, we introduce a new framework, Reward rAnked FineTuning (RAFT), designed to align generative models effectively. Utilizing a reward model and a sufficient number of samples, our approach selects the high-quality samples, discarding those that exhibit undesired behavior, and subsequently enhancing the model by fine-tuning on these filtered samples. Our studies show that RAFT can effectively improve the model performance in both reward learning and other automated metrics in both large language models and diffusion models.

Key facts. Cited by 686 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 60 of its 81 references that are in the map. First posted on arXiv 13 Apr 2023.

Builds on

60 of 81 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  2. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  10. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  11. 2020Denoising Diffusion Implicit Modelscited by 9,781
  12. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
38 more
  1. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2022Emergent Abilities of Large Language Modelscited by 2,512
  13. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  14. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  15. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  16. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  17. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  18. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  19. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  20. 2018Hierarchical Neural Story Generationcited by 1,193
  21. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  22. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  23. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  24. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 672
  25. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 652
  26. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  27. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  28. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  29. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  30. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  31. 2018AI safety via debatecited by 400
  32. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  33. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  34. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preferencecited by 287
  35. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  36. 2022A Contrastive Framework for Neural Text Generationcited by 225
  37. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 221
  38. 2022Optimizing Prompts for Text-to-Image Generationcited by 207

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 85; 2024: 236; 2025: 244; 2026: 121.

686 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  4. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
  5. 2023A Comprehensive Overview of Large Language Modelscited by 648
  6. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  7. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  8. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  9. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  10. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  11. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 448
  12. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
38 more
  1. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  2. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  3. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  4. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  5. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  6. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  7. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  8. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
  9. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 288
  10. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  11. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  12. 2025Improving Video Generation with Human Feedbackcited by 284
  13. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  14. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  15. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 260
  16. 2025SAM 3D: 3Dfy Anything in Imagescited by 247
  17. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  18. 2023Large Language Model Alignment: A Surveycited by 243
  19. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  20. 2023Nash Learning from Human Feedbackcited by 224
  21. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  22. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 200
  23. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  24. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  25. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 183
  26. 2023Mitigating the Alignment Tax of RLHFcited by 183
  27. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  28. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  29. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
  30. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  31. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  32. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 166
  33. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 160
  34. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 157
  35. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 148
  36. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 145
  37. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  38. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 136

50 most cited of 686 All 686 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Long-form RewardBench: Evaluating Reward Models for Long-form Generation
  2. 2026Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models
  3. 2025GRAM: A Generative Foundation Reward Model for Reward Generalization
  4. 2026CRAFT: Aligning Diffusion Models with Fine-Tuning Is Easier Than You Think
  5. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  6. 2025Generative RLHF-V: Learning Principles from Multi-modal Human Preference
  7. 2024Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment
  8. 2024SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins