mlsanityData to 7 Oct 2026

SoFA: Shielded On-the-fly Alignment via Priority Rule Following

arXiv 2402.17358 ↗First posted 27 Feb 2024cs.CLcited by 18

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The alignment problem in Large Language Models (LLMs) involves adapting them to the broad spectrum of human values. This requirement challenges existing alignment methods due to diversity of preferences and regulatory standards. This paper introduces a novel alignment paradigm, priority rule following, which defines rules as the primary control mechanism in each dialog, prioritizing them over user instructions. Our preliminary analysis reveals that even the advanced LLMs, such as GPT-4, exhibit shortcomings in understanding and prioritizing the rules. Therefore, we present PriorityDistill, a semi-automated approach for distilling priority following signals from LLM simulations to ensure robust rule integration and adherence. Our experiments show that this method not only effectively minimizes misalignments utilizing only one general rule but also adapts smoothly to various unseen rules, ensuring they are shielded from hijacking and that the model responds appropriately.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 53 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

33 of 53 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Large Language Model Alignment: A Survey26 Sep
  2. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  4. 2022Training language models to follow instructions with human feedback4 Mar
  5. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 28

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  9. 2022Red Teaming Language Models with Language Modelscited by 983
  10. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  11. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  12. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
16 more
  1. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  2. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  3. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  4. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  5. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  6. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  7. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  8. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  9. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  10. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
  11. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  12. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  13. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 135
  14. 2023Can LLMs Follow Simple Rules?cited by 45
  15. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 42
  16. 2023Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulationcited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 8; 2026: 2.

18 papers.

Sort:

By area:

  1. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  2. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 80
  3. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 67
  4. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 55
  5. 2025IHEval: Evaluating Language Models on Following the Instruction Hierarchycited by 48
  6. 2025A Closer Look at System Prompt Robustnesscited by 31
  7. 2024SysBench: Can Large Language Models Follow System Messages?cited by 26
  8. 2025X-Boundary: Establishing Exact Safety Boundary to Shield LLMs from Multi-Turn Jailbreaks without Compromising Usabilitycited by 20
  9. 2024CantTalkAboutThis: Aligning Language Models to Stay on Topic in Dialoguescited by 15
  10. 2025LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-Disjointcited by 7
  11. 2024Baichuan Alignment Technical Reportcited by 6
  12. 2025LifeAlign: Lifelong Alignment for Large Language Models with Memory-Augmented Focalized Preference Optimizationcited by 6
6 more
  1. 2024RuleAlign: Making Large Language Models Better Physicians with Diagnostic Rule Alignmentcited by 3
  2. 2025Rule-Guided Feedback: Enhancing Reasoning by Enforcing Rule Adherence in Large Language Modelscited by 3
  3. 2025Learning to Route: A Rule-Driven Agent Framework for Hybrid-Source Retrieval-Augmented Generationcited by 3
  4. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  5. 2025System Message Generation for User Preferences using Open-Source Modelscited by 1
  6. 2026Compliance, Capability, and Conflict: Benchmarking Multimodal LLMs under System Messagescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  2. 2025P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesis
  3. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment
  4. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
  5. 2023CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment
  6. 2026ECLIPTICA -- A Framework for Switchable LLM Alignment via CITA - Contrastive Instruction-Tuned Alignment
  7. 2026SpecAlign: Efficient Specification-Grounded Alignment of Large Language Models via Synthetic Data
  8. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications