mlsanityData to 7 Oct 2026

SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF

arXiv 2310.05344 ↗First posted 9 Oct 2023cs.CL cs.AI cs.LGcited by 107

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Model alignment with human preferences is an essential step in making Large Language Models (LLMs) helpful and consistent with human values. It typically consists of supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) stages. However, RLHF faces inherent limitations stemming from a complex training setup and its tendency to align the model with implicit values that end users cannot control at run-time. Moreover, reward models in RLHF stage commonly rely on single-dimensional feedback as opposed to explicit, multifaceted signals that indicate attributes such as helpfulness, humor, and toxicity. To address these limitations, we propose SteerLM, a supervised fine-tuning method that empowers end-users to control responses during inference. SteerLM conditions responses to conform to an explicitly defined multi-dimensional set of attributes, thereby empowering a steerable AI capable of generating helpful and high-quality responses while maintaining customizability. Experiments show that SteerLM trained on open source datasets generates responses that are preferred by human and automatic evaluators to many state-of-the-art baselines trained with RLHF while being much easier to train. Try SteerLM at https://huggingface.co/nvidia/SteerLM-llama2-13B

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 47 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

31 of 47 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  3. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  5. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
13 more
  1. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  2. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  3. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  4. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  5. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  6. 2023Instruction Tuning with GPT-4cited by 676
  7. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  8. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 230
  9. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  10. 2020Controlling Style in Generated Dialoguecited by 38
  11. 2023ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Modelscited by 23
  12. 2021Extracting and Inferring Personal Attributes from Dialoguecited by 8
  13. 2022Measuring and signing fairness as performance under multiple stakeholder distributionscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 42; 2025: 30; 2026: 31.

107 papers.

Sort:

By area:

  1. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  2. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  3. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 168
  4. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  5. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
  6. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 126
  7. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 105
  8. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  9. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
  10. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  11. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 78
  12. 2023Data Diversity Matters for Robust Instruction Tuningcited by 76
38 more
  1. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
  2. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  3. 2023Can LLMs Follow Simple Rules?cited by 45
  4. 2025Multi-Attribute Steering of Language Models via Targeted Interventioncited by 44
  5. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 40
  6. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  7. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 40
  8. 2024CodecLM: Aligning Language Models with Tailored Synthetic Datacited by 39
  9. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 35
  10. 2024Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Modelscited by 27
  11. 2024Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Modelscited by 26
  12. 2024A Survey on Human Preference Learning for Large Language Modelscited by 23
  13. 2024Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Modelscited by 22
  14. 2025Robust Multi-Objective Preference Alignment with Online DPOcited by 21
  15. 2024Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)cited by 20
  16. 2025AI Alignment at Your Discretioncited by 20
  17. 2024Learning Code Preference via Synthetic Evolutioncited by 19
  18. 2025Understanding and Mitigating Overrefusal in LLMs from an Unveiling Perspective of Safety Decision Boundarycited by 17
  19. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  20. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  21. 2024Language Models are Alignable Decision-Makers: Dataset and Application to the Medical Triage Domaincited by 13
  22. 2024Identifying and Manipulating Personality Traits in LLMs Through Activation Engineeringcited by 13
  23. 2025Bone Soups: A Seek-and-Soup Model Merging Approach for Controllable Multi-Objective Generationcited by 13
  24. 2025Ask a Strong LLM Judge when Your Reward Model is Uncertaincited by 13
  25. 2024Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Modelscited by 12
  26. 2024Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Modelscited by 11
  27. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMscited by 10
  28. 2024On the Loss of Context-awareness in General Instruction Fine-tuningcited by 10
  29. 2025BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoningcited by 10
  30. 2025CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectivescited by 10
  31. 2024A Preference-driven Paradigm for Enhanced Translation with Large Language Modelscited by 9
  32. 2024Interpreting Language Reward Models via Contrastive Explanationscited by 9
  33. 2025MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learningcited by 9
  34. 2024Steering Conversational Large Language Models for Long Emotional Support Conversationscited by 8
  35. 2024SudoLM: Learning Access Control of Parametric Knowledge with Authorization Alignmentcited by 8
  36. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 8
  37. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 8
  38. 2024OpenLLM-Ro -- Technical Report on Open-source Romanian LLMscited by 7

50 most cited of 107 All 107 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes
  2. 2023SuperHF: Supervised Iterative Learning from Human Feedback
  3. 2026The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement
  4. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  5. 2024Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads
  6. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback
  7. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  8. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback