mlsanityData to 7 Oct 2026

Pretraining Language Models with Human Preferences

arXiv 2302.08582 ↗First posted 16 Feb 2023cs.CL cs.LGcited by 245

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Language models (LMs) are pretrained to imitate internet text, including content that would violate human preferences if generated by an LM: falsehoods, offensive comments, personally identifiable information, low-quality or buggy code, and more. Here, we explore alternative objectives for pretraining LMs in a way that also guides them to generate text aligned with human preferences. We benchmark five objectives for pretraining with human feedback across three tasks and study how they affect the trade-off between alignment and capabilities of pretrained LMs. We find a Pareto-optimal and simple approach among those we explored: conditional training, or learning distribution over tokens conditional on their human preference scores given by a reward model. Conditional training reduces the rate of undesirable content by up to an order of magnitude, both when generating without a prompt and with an adversarially-chosen prompt. Moreover, conditional training maintains the downstream task performance of standard LM pretraining, both before and after task-specific finetuning. Pretraining with human feedback results in much better preference satisfaction than standard LM pretraining followed by finetuning with feedback, i.e., learning and then unlearning undesirable behavior. Our results suggest that we should move beyond imitation learning when pretraining LMs and incorporate human preferences from the start of training.

Key facts. Cited by 245 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 62 of its 98 references that are in the map. First posted on arXiv 16 Feb 2023.

Builds on

62 of 98 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training Language Models with Language Feedback29 Apr

Other references · 61

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
38 more
  1. 2020Extracting Training Data from Large Language Modelscited by 2,144
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  8. 2018Hierarchical Neural Story Generationcited by 1,182
  9. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  10. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  11. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  12. 2022Red Teaming Language Models with Language Modelscited by 983
  13. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  14. 2018Neural Network Acceptability Judgmentscited by 952
  15. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  16. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  17. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  18. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  19. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  20. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  21. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 511
  22. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  23. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  24. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
  25. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  26. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  27. 2022Teaching language models to support answers with verified quotescited by 290
  28. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  29. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  30. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 187
  31. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  32. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  33. 2021Challenges in Detoxifying Language Modelscited by 169
  34. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  35. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  36. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 147
  37. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  38. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 67; 2024: 94; 2025: 54; 2026: 30.

245 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  3. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  4. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  6. 2023PaLM 2 Technical Reportcited by 1,154
  7. 2023Teaching Large Language Models to Self-Debugcited by 963
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  11. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  12. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
38 more
  1. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  2. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  3. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  4. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  5. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  6. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  7. 2023Challenges and Applications of Large Language Modelscited by 251
  8. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  9. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  10. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  11. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  12. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  13. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  14. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  15. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  16. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  17. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  18. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 173
  19. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  20. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  21. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  22. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 128
  23. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  24. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  25. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  26. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  27. 2023Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Casescited by 85
  28. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  29. 2023Aligning Large Language Models through Synthetic Feedbackcited by 75
  30. 2023Eight Things to Know about Large Language Modelscited by 73
  31. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  32. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  33. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 70
  34. 2024Can LLMs Learn from Previous Mistakes? Investigating LLMs' Errors to Boost for Reasoningcited by 69
  35. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  36. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 67
  37. 2024Intention Analysis Makes LLMs A Good Jailbreak Defendercited by 64
  38. 2023Jailbreaking GPT-4V via Self-Adversarial Attacks with System Promptscited by 62

50 most cited of 245 All 245 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesis
  2. 2024A Survey on Human Preference Learning for Large Language Models
  3. 2023Training Language Models with Language Feedback at Scale
  4. 2023Aligning Language Models with Offline Learning from Human Feedback
  5. 2024A Systematic Examination of Preference Learning through the Lens of Instruction-Following
  6. 2024Dissecting Human and LLM Preferences
  7. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback
  8. 2024Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments