mlsanityData to 7 Oct 2026

Safer-Instruct: Aligning Language Models with Automated Preference Data

arXiv 2311.08685 ↗First posted 15 Nov 2023cs.CL cs.AIcited by 27

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) is a vital strategy for enhancing model capability in language models. However, annotating preference data for RLHF is a resource-intensive and creativity-demanding process, while existing automatic generation methods face limitations in data diversity and quality. In response, we present Safer-Instruct, a novel pipeline for automatically constructing large-scale preference data. Our approach leverages reversed instruction tuning, instruction induction, and expert model evaluation to efficiently generate high-quality preference data without human annotators. To verify the effectiveness of Safer-Instruct, we apply the pipeline to construct a safety preference dataset as a case study. Finetuning an Alpaca model on this synthetic dataset not only demonstrates improved harmlessness but also outperforms models fine-tuned on human-annotated safety preference data, all the while maintaining a competitive edge in downstream tasks. Importantly, our Safer-Instruct framework is versatile and can be applied to generate preference data across various domains, extending its utility beyond safety preferences. It addresses the challenges in preference data acquisition and advances the development of more capable and responsible AI systems. For dataset and code implementation, see https://github.com/uscnlp-lime/safer-instruct

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 56 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

31 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  2. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
16 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  3. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  4. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  6. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  7. 2018QuAC : Question Answering in Contextcited by 379
  8. 2020The Pushshift Reddit Datasetcited by 289
  9. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  10. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  11. 2020Directions in Abusive Language Training Data: Garbage In, Garbage Outcited by 139
  12. 2020Measuring a hate speech spectrum with faceted Rasch item response theory and perspective-aware, explainable-by-design deep learningcited by 67
  13. 2021Deep Learning for Suicide and Depression Identification with Unsupervised Label Correctioncited by 45
  14. 2023How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization?cited by 43
  15. 2021SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failurescited by 37
  16. 2022Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learnerscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 12; 2026: 5.

27 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  2. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  3. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  4. 2024Self-Boosting Large Language Models with Synthetic Preference Datacited by 40
  5. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 35
  6. 2024WildFeedback: Aligning LLMs With In-situ User Interactions And Feedbackcited by 31
  7. 2024How Susceptible are Large Language Models to Ideological Manipulation?cited by 26
  8. 2026Experiential Reinforcement Learningcited by 21
  9. 2024Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?cited by 15
  10. 2024Verifiable by Design: Aligning Language Models to Quote from Pre-Training Datacited by 13
  11. 2024Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Modelscited by 13
  12. 2025An Embarrassingly Simple Defense Against LLM Abliteration Attackscited by 12
15 more
  1. 2024SafeWorld: Geo-Diverse Safety Alignmentcited by 10
  2. 2025Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attackscited by 7
  3. 2025Discovering Knowledge Deficiencies of Language Models on Massive Knowledge Basecited by 6
  4. 2025ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Datacited by 6
  5. 2025STEER-BENCH: A Benchmark for Evaluating the Steerability of Large Language Modelscited by 6
  6. 2024The Real, the Better: Aligning Large Language Models with Online Human Behaviorscited by 3
  7. 2025Semi-Supervised Preference Optimization with Limited Feedbackcited by 3
  8. 2026DUET: Distilled LLM Unlearning from an Efficiently Contextualized Teachercited by 3
  9. 2026GEM-Occ: From Visual Geometry Evidence to Embodied Semantic Occupancy Memorycited by 3
  10. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  11. 2025PanicToCalm: A Proactive Counseling Agent for Panic Attackscited by 1
  12. 2025ZEBRA: Leveraging Model-Behavioral Knowledge for Zero-Annotation Preference Dataset Constructioncited by 0
  13. 2025RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Modelscited by 0
  14. 2025From Threat to Tool: Leveraging Refusal-Aware Injection Attacks for Safety Alignmentcited by 0
  15. 2026Generating Place-Based Compromises Between Two Points of Viewcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages
  2. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignment
  3. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety
  4. 2024Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness
  5. 2025Reinforcement Learning from Human Feedback with High-Confidence Safety Constraints
  6. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  7. 2025Data-adaptive Safety Rules for Training Reward Models
  8. 2025Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Models