mlsanityData to 13 Oct 2026

SLiC-HF: Sequence Likelihood Calibration with Human Feedback

arXiv 2305.10425 ↗First posted 17 May 2023cs.CL cs.AIcited by 400

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Learning from human feedback has been shown to be effective at aligning language models with human preferences. Past work has often relied on Reinforcement Learning from Human Feedback (RLHF), which optimizes the language model using reward scores assigned from a reward model trained on human preference data. In this work we show how the recently introduced Sequence Likelihood Calibration (SLiC), can also be used to effectively learn from human preferences (SLiC-HF). Furthermore, we demonstrate this can be done with human feedback data collected for a different model, similar to off-policy, offline RL data. Automatic and human evaluation experiments on the TL;DR summarization task show that SLiC-HF significantly improves supervised fine-tuning baselines. Furthermore, SLiC-HF presents a competitive alternative to the PPO RLHF implementation used in past work while being much simpler to implement, easier to tune and more computationally efficient in practice.

Key facts. Cited by 400 papers in the map (data through 13 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 22 references that are in the map. First posted on arXiv 17 May 2023.

Builds on

19 of 22 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  2. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2022Training language models to follow instructions with human feedbackcited by 15,856
  3. 2017Proximal Policy Optimization Algorithmscited by 14,592
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,011
  7. 2020Learning to summarize from human feedbackcited by 2,575
  8. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,473
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,041
  10. 2022Improving alignment of dialogue agents via targeted human judgementscited by 529
  11. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  12. 2022News Summarization and Evaluation in the Era of GPT-3cited by 326
5 more
  1. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  2. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  3. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  4. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 101
  5. 2022mFACE: Multilingual Summarization with Factual Consistency Evaluationcited by 42

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 37; 2024: 191; 2025: 126; 2026: 46.

400 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,020
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 954
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 909
  4. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 775
  5. 2024Self-Rewarding Language Modelscited by 666
  6. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 371
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 347
  8. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 322
  9. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 297
  10. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 287
  11. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  12. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 261
38 more
  1. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 259
  2. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  3. 2023Large Language Model Alignment: A Surveycited by 243
  4. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  5. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
  6. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 229
  7. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 226
  8. 2023Nash Learning from Human Feedbackcited by 223
  9. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 203
  10. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 196
  11. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  12. 2023Mitigating the Alignment Tax of RLHFcited by 183
  13. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearningcited by 167
  14. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 156
  15. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 148
  16. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 143
  17. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 142
  18. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 142
  19. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 134
  20. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  21. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  22. 2024Direct Preference Optimization with an Offsetcited by 122
  23. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
  24. 2024Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimizationcited by 114
  25. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  26. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  27. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  28. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  29. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 108
  30. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 107
  31. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  32. 2024Critique-out-Loud Reward Modelscited by 105
  33. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 102
  34. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  35. 2024Aligning Diffusion Models by Optimizing Human Utilitycited by 100
  36. 2024LiPO: Listwise Preference Optimization through Learning-to-Rankcited by 96
  37. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  38. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95

50 most cited of 400 All 400 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  2. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears
  5. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
  6. 2024How to Evaluate Reward Models for RLHF
  7. 2024Towards Comprehensive Preference Data Collection for Reward Modeling
  8. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback