mlsanityData to 14 Oct 2026

Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models

arXiv 2408.15313 ↗First posted 27 Aug 2024cs.AI cs.CL cs.LGcited by 27

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Fine-tuning large language models (LLMs) on human preferences, typically through reinforcement learning from human feedback (RLHF), has proven successful in enhancing their capabilities. However, ensuring the safety of LLMs during fine-tuning remains a critical concern, and mitigating the potential conflicts in safety and helpfulness is costly in RLHF. To address this issue, we propose a supervised learning framework called Bi-Factorial Preference Optimization (BFPO), which re-parameterizes a joint RLHF objective of both safety and helpfulness into a single supervised learning objective. In supervised optimization, a labeling function is used to capture the global preferences ranking to balance both safety and helpfulness. To evaluate BFPO, we develop a benchmark that includes comprehensive discriminative and generative tasks for helpfulness and harmlessness. The results indicate that our method significantly outperforms existing approaches in both safety and helpfulness. Moreover, BFPO achieves the same level of safety as methods that heavily rely on human labor with less than 10% of the computational resources and human prompting and annotation process. The training recipes can be found here: https://github.com/wx-zhang/bfpo.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 60 references that are in the map. First posted on arXiv 27 Aug 2024.

Builds on

48 of 60 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  2. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  3. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  5. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct
  6. 2022Training language models to follow instructions with human feedback4 Mar
  7. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 41

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023Mistral 7Bcited by 3,164
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2021Convex optimizationcited by 2,515
  11. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
29 more
  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  3. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  4. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  6. 2021Ethical and social risks of harm from Language Modelscited by 966
  7. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  8. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  11. 2020Aligning AI With Shared Human Valuescited by 723
  12. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  13. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  14. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  15. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  16. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  17. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  18. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  19. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 312
  20. 2019On Tiny Episodic Memories in Continual Learningcited by 307
  21. 2021Scaling Laws for Transfercited by 272
  22. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  23. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
  24. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123
  25. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 111
  26. 2020Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotescited by 101
  27. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 98
  28. 2023Overcoming Generic Knowledge Loss with Selective Parameter Updatecited by 23
  29. 2024Benchmarking Llama2, Mistral, Gemma and GPT for Factuality, Toxicity, Bias and Propensity for Hallucinationscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 17; 2026: 8.

27 papers.

Sort:

By area:

  1. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  2. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 134
  3. 2025Multi-Attribute Steering of Language Models via Targeted Interventioncited by 46
  4. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 37
  5. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Modelscited by 23
  6. 2025Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Modelscited by 22
  7. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  8. 2025Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignmentcited by 16
  9. 2025Early External Safety Testing of OpenAI's o3-mini: Insights from the Pre-Deployment Evaluationcited by 13
  10. 2025Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Modelscited by 12
  11. 2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
  12. 2025Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimizationcited by 11
15 more
  1. 2025ASTRAL: Automated Safety Testing of Large Language Modelscited by 7
  2. 2025Adaptive Helpfulness-Harmlessness Alignment with Preference Vectorscited by 6
  3. 2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
  4. 2025When Style Breaks Safety: Defending LLMs Against Superficial Style Alignmentcited by 6
  5. 2025A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Userscited by 6
  6. 2025Reinforcement Learning from Human Feedback with High-Confidence Safety Constraintscited by 4
  7. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
  8. 2026SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignmentcited by 4
  9. 2025MidPO: Dual Preference Optimization for Safety and Helpfulness in Large Language Models via a Mixture of Experts Frameworkcited by 3
  10. 2026dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Modelscited by 3
  11. 2026Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignmentcited by 2
  12. 2026Please refuse to answer me! Mitigating Over-Refusal in Large Language Models via Adaptive Contrastive Decodingcited by 2
  13. 2026Dialectics of Alignment: Harnessing Unsafe Knowledge for Dynamic Safety Routingcited by 0
  14. 2026ReFrame: Evidence-Guided Test-Time Safety Alignment in Multimodal Large Language Modelscited by 0
  15. 2026MOAE: Multi-Objective Agent Evolution with Pareto-Preserving Searchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolation
  2. 2024Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness
  3. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety
  4. 2025A Survey of Direct Preference Optimization
  5. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  6. 2024New Desiderata for Direct Preference Optimization
  7. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  8. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences