FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization
On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods
Abstract (arXiv)
Recent breakthroughs in preference alignment have significantly improved Large Language Models' ability to generate texts that align with human preferences and values. However, current alignment metrics typically emphasize the post-hoc overall improvement, while overlooking a critical aspect: regression, which refers to the backsliding on previously correctly-handled data after updates. This potential pitfall may arise from excessive fine-tuning on already well-aligned data, which subsequently leads to over-alignment and degeneration. To address this challenge, we propose FlipGuard, a constrained optimization approach to detect and mitigate update regression with focal attention. Specifically, FlipGuard identifies performance degradation using a customized reward characterization and strategically enforces a constraint to encourage conditional congruence with the pre-aligned model during training. Comprehensive experiments demonstrate that FlipGuard effectively alleviates update regression while demonstrating excellent overall performance, with the added benefit of knowledge preservation while aligning preferences.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 51 references that are in the map. First posted on arXiv 1 Oct 2024.
Builds on
40 of 51 references.
Closest ancestors · 9
Same direction, newest first
- 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment28 May
- 2024LIRE: listwise reward enhancement for preference alignment22 May
- 2024Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction22 May
- 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
- 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
- 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning4 Dec
- 2023Mitigating the Alignment Tax of RLHF12 Sep
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback24 May
- 2022Training language models to follow instructions with human feedback4 Mar
Other references · 31
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
19 more
- 2020Learning to summarize from human feedbackcited by 2,562
- 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
- 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
- 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
- 2021Alignment of Language Agentscited by 167
- 2021Focal and Global Knowledge Distillation for Detectorscited by 138
- 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
- 2020A Distributional Approach to Controlled Text Generationcited by 117
- 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 107
- 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
- 2023CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibilitycited by 80
- 2020Towards Backward-Compatible Representation Learningcited by 74
- 2019Toward Understanding Catastrophic Forgetting in Continual Learningcited by 67
- 2020Positive-Congruent Training: Towards Regression-Free Model Updatescited by 55
- 2023Reinforcement Learning from Diverse Human Preferencescited by 24
- 2018Pseudo-Recursal: Solving the Catastrophic Forgetting Problem in Deep Neural Networkscited by 23
- 2021Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updatescited by 13
- 2022Measuring and Reducing Model Update Regression in Structured Prediction for NLPcited by 11
- 2023Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusioncited by 6
Built on by
Citing papers per year: 2025: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
- 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment
- 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization
- 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
- 2025When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHF
- 2026An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shift
- 2025Towards Reliable, Uncertainty-Aware Alignment
- 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization