mlsanityData to 7 Oct 2026

BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization

arXiv 2407.00693 ↗First posted 30 Jun 2024cs.AI cs.CL cs.LGcited by 6

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

While learning to align Large Language Models (LLMs) with human preferences has shown remarkable success, aligning these models to meet the diverse user preferences presents further challenges in preserving previous knowledge. This paper examines the impact of personalized preference optimization on LLMs, revealing that the extent of knowledge loss varies significantly with preference heterogeneity. Although previous approaches have utilized the KL constraint between the reference model and the policy model, we observe that they fail to maintain general knowledge and alignment when facing personalized preferences. To this end, we introduce Base-Anchored Preference Optimization (BAPO), a simple yet effective approach that utilizes the initial responses of reference model to mitigate forgetting while accommodating personalized alignment. BAPO effectively adapts to diverse user preferences while minimally affecting global knowledge or general alignment. Our experiments demonstrate the efficacy of BAPO in various setups.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 56 references that are in the map. First posted on arXiv 30 Jun 2024.

Builds on

43 of 56 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment28 May
  2. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment29 Feb
  3. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  4. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  5. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  6. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs3 Feb
  7. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  8. 2023On Diversified Preferences of Large Language Model Alignment12 Dec
  9. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  11. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  12. 2023Making Large Language Models Better Reasoners with Alignment5 Sep
2 more
  1. 2022Training language models to follow instructions with human feedback4 Mar
  2. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 29

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2020Learning to summarize from human feedbackcited by 2,562
17 more
  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  4. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  5. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  6. 2024LoRA Learns Less and Forgets Lesscited by 313
  7. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  8. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  9. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  10. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  11. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  12. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 124
  13. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  14. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  15. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 42
  16. 2024MORL-Prompt: An Empirical Analysis of Multi-Objective Reinforcement Learning for Discrete Prompt Optimizationcited by 16
  17. 2024What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinementcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistantscited by 44
  2. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 25
  3. 2025Aligning VLM Assistants with Personalized Situated Cognitioncited by 2
  4. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  5. 2026SAGA: Score-Weighted Adaptive Generation Alignment for Low-Resource Nordic Language Modelscited by 0
  6. 2026StalePO: Anchored Token-Level Preference Optimization using Legacy Post-Edits in Machine Translationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025InfoPO: On Mutual Information Maximization for Large Language Model Alignment
  2. 2025LifeAlign: Lifelong Alignment for Large Language Models with Memory-Augmented Focalized Preference Optimization
  3. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  4. 2024Multi-Reference Preference Optimization for Large Language Models
  5. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  6. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  7. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications
  8. 2026Learning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning