mlsanityData to 7 Oct 2026

Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment

arXiv 2405.17888 ↗First posted 28 May 2024cs.AIcited by 32

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Aligning human preference and value is an important requirement for contemporary foundation models. State-of-the-art techniques such as Reinforcement Learning from Human Feedback (RLHF) often consist of two stages: 1) supervised fine-tuning (SFT), where the model is fine-tuned by learning from human demonstration data; 2) Preference learning, where preference data is used to learn a reward model, which is in turn used by a reinforcement learning (RL) step to fine-tune the model. Such reward model serves as a proxy to human preference, and it is critical to guide the RL step towards improving the model quality. In this work, we argue that the SFT stage significantly benefits from learning a reward model as well. Instead of using the human demonstration data directly via supervised learning, we propose to leverage an Inverse Reinforcement Learning (IRL) technique to simultaneously build an reward model and a policy model. This approach leads to new SFT algorithms that are not only efficient to implement, but are robust to the presence of low-quality supervised learning data. Moreover, we discover a connection between the proposed IRL based approach, and a recent line of works called Self-Play Fine-tune (SPIN). Theoretically, we show that the proposed algorithms converge to the stationary solutions of the IRL problem. Empirically, we align 1B and 7B models using proposed methods and evaluate them on a reward benchmark model and the HuggingFace Open LLM Leaderboard. The proposed methods show significant performance improvement over existing SFT approaches. Our results indicate that it is beneficial to leverage reward learning throughout the entire alignment process.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 57 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

40 of 57 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Self-Rewarding Language Models18 Jan
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  4. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2021Program Synthesis with Large Language Modelscited by 3,509
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
23 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  10. 2023PaLM 2 Technical Reportcited by 1,154
  11. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  13. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  14. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  15. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  16. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  17. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  18. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  19. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  20. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  21. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  22. 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guaranteescited by 42
  23. 2022Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guaranteescited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 20; 2026: 8.

32 papers.

Sort:

By area:

  1. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  2. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  3. 2025LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Callingcited by 58
  4. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoningcited by 28
  5. 2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
  6. 2025Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignmentcited by 20
  7. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  8. 2026ShowUI-Aloha: Human-Taught GUI Agentcited by 7
  9. 2025Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Models Using Implicit Feedback from Pre-training Demonstrationscited by 6
  10. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 6
  11. 2025RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Modelscited by 5
  12. 2025ALTo: Adaptive-Length Tokenizer for Autoregressive Mask Generationcited by 5
20 more
  1. 2025CrowdLLM: Building LLM-Based Digital Populations Augmented with Generative Modelscited by 5
  2. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferencescited by 4
  3. 2025Automatic MILP Model Construction for Multi-Robot Task Allocation and Scheduling Based on Large Language Modelscited by 3
  4. 2025We Think, Therefore We Align LLMs to Helpful, Harmless and Honest Before They Go Wrongcited by 3
  5. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  6. 2025Beyond External Monitors: Enhancing Transparency of Large Language Models for Easier Monitoringcited by 2
  7. 2026Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weightingcited by 2
  8. 2024Enhancing Spectral Knowledge Interrogation: A Reliable Retrieval-Augmented Generative Framework on Large Language Modelscited by 1
  9. 2025Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Datacited by 1
  10. 2025InvestAlign: Overcoming Data Scarcity in Aligning Large Language Models with Investor Decision-Making Processes under Herd Behaviorcited by 1
  11. 2025Beyond Imitation: Recovering Dense Rewards from Demonstrationscited by 1
  12. 2025Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Onlycited by 1
  13. 2026Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environmentscited by 1
  14. 2025DMRL: Data- and Model-aware Reward Learning for Data Extractioncited by 0
  15. 2025Datarus-R1: An Adaptive Multi-Step Reasoning LLM for Automated Data Analysiscited by 0
  16. 2025Nonconvex Decentralized Stochastic Bilevel Optimization under Heavy-Tailed Noisecited by 0
  17. 2026Learn from Your Mistakes: Tree-like Self-Play for Secure Code LLMscited by 0
  18. 2026SCOUT: Self-Checking and Recovery-Aware Tool-Thought Agents for Ultra-Long Egocentric Video Reasoningcited by 0
  19. 2026To Solve Bilevel Optimization with Nonconvex Lower Levels, We Need Second-Order Stationaritycited by 0
  20. 2026Learning the Robustness Mechanism with Bilevel Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignment
  2. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  3. 2024Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning
  4. 2024Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment
  5. 2025RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment
  6. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  7. 2025Influencing Humans to Conform to Preference Models for RLHF
  8. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning