mlsanityData to 7 Oct 2026

SALMON: Self-Alignment with Instructable Reward Models

arXiv 2310.05910 ↗First posted 9 Oct 2023cs.CL cs.AI cs.LGcited by 54

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Supervised Fine-Tuning (SFT) on response demonstrations combined with Reinforcement Learning from Human Feedback (RLHF) constitutes a powerful paradigm for aligning LLM-based AI agents. However, a significant limitation of such an approach is its dependency on high-quality human annotations, making its application to intricate tasks challenging due to difficulties in obtaining consistent response demonstrations and in-distribution response preferences. This paper presents a novel approach, namely SALMON, to align base language models with minimal human supervision, using only a small set of human-defined principles, yet achieving superior performance. Central to our approach is an instructable reward model. Trained on synthetic preference data, this model can generate reward scores based on arbitrary human-defined principles. By merely adjusting these principles during the RL training phase, we gain full control over the preferences with the instructable reward model, subsequently influencing the behavior of the RL-trained policy models, and reducing the reliance on the collection of online human preferences. Applying our method to the LLaMA-2-70b base language model, we developed an AI assistant named Dromedary-2. With only 6 exemplars for in-context learning and 31 human-defined principles, Dromedary-2 significantly surpasses the performance of several state-of-the-art AI systems, including LLaMA-2-Chat-70b, on various benchmark datasets. We have open-sourced the code and model weights to encourage further research into aligning LLM-based AI agents with enhanced supervision efficiency, improved controllability, and scalable oversight.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 65 of its 137 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

65 of 137 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment24 Jul
  2. 2023Aligning Large Language Models through Synthetic Feedback23 May
  3. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  4. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  6. 2022Training language models to follow instructions with human feedback4 Mar
  7. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 58

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  12. 2017Deep reinforcement learning from human preferencescited by 3,846
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  12. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  13. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  14. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  15. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  16. 2016Concrete Problems in AI Safetycited by 1,582
  17. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  18. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  19. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  20. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  21. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  22. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  23. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  24. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  25. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  26. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  27. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  28. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  29. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  30. 2020Artificial Intelligence, Values and Alignmentcited by 443
  31. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  32. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  33. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  34. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  35. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  36. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  37. 2023The False Promise of Imitating Proprietary LLMscited by 254
  38. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 24; 2025: 18; 2026: 10.

54 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  3. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  4. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  5. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 108
  6. 2025Reinforcement Learning with Rubric Anchorscited by 108
  7. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  8. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  9. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 61
  10. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  11. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  12. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
38 more
  1. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 49
  2. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  3. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  4. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  5. 2024Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedbackcited by 36
  6. 2025Beyond the Surface: Measuring Self-Preference in LLM Judgmentscited by 30
  7. 2025QA-LIGN: Aligning LLMs through Constitutionally Decomposed QAcited by 28
  8. 2023Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Gamecited by 27
  9. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  10. 2024Human-Instruction-Free LLM Self-Alignment with Limited Samplescited by 25
  11. 2024Weaver: Foundation Models for Creative Writingcited by 25
  12. 2024ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedbackcited by 25
  13. 2024MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimizationcited by 20
  14. 2024Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrappingcited by 14
  15. 2025Aligning LLMs by Predicting Preferences from User Writing Samplescited by 11
  16. 2025BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Datasetcited by 11
  17. 2024Varying Shades of Wrong: Aligning LLMs with Wrong Answers Onlycited by 9
  18. 2024Rethinking the Role of Proxy Rewards in Language Model Alignmentcited by 7
  19. 2024Fictitious Synthetic Data Can Improve LLM Factuality via Prerequisite Learningcited by 7
  20. 2024Towards Full Delegation: Designing Ideal Agentic Behaviors for Travel Planningcited by 7
  21. 2024VideoSAVi: Self-Aligned Video Language Models without Human Supervisioncited by 7
  22. 2025Think Twice: Branch-and-Rethink Reasoning Reward Modelcited by 7
  23. 2024FastAdaSP: Multitask-Adapted Efficient Inference for Large Speech Language Modelcited by 5
  24. 2024Preference-Guided Reflective Sampling for Aligning Language Modelscited by 4
  25. 2025RankAlign: A Ranking View of the Generator-Validator Gap in Large Language Modelscited by 4
  26. 2026Unifying Stable Optimization and Reference Regularization in RLHFcited by 4
  27. 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategycited by 4
  28. 2025EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferencescited by 3
  29. 2025Towards Faithful and Controllable Personalization via Critique-Post-Edit Reinforcement Learningcited by 3
  30. 2026Beyond Verifiable Rewards: Rubric-Based GRM for Reinforced Fine-Tuning SWE Agentscited by 3
  31. 2025RetroMotion: Retrocausal Motion Forecasting Models are Instructablecited by 2
  32. 2024Solving the Inverse Alignment Problem for Efficient RLHFcited by 1
  33. 2025QM-ToT: A Medical Tree of Thoughts Reasoning Framework for Quantized Modelcited by 1
  34. 2026Generating Data-Driven Reasoning Rubrics for Domain-Adaptive Reward Modelingcited by 1
  35. 2026Utilizing and Calibrating Hindsight Process Rewards via Reinforcement with Mutual Information Self-Evaluationcited by 1
  36. 2025Latent Principle Discovery for Language Model Self-Improvementcited by 0
  37. 2025Factors That Support Grounded Responses in LLM Conversations: A Rapid Reviewcited by 0
  38. 2026CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judgingcited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignment
  2. 2025RLTHF: Targeted Human Feedback for LLM Alignment
  3. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  4. 2025Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only
  5. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  6. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment
  7. 2024Direct Alignment of Language Models via Quality-Aware Self-Refinement
  8. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models