mlsanityData to 14 Oct 2026

Leveraging Machine Unlearning for Cost-Efficient Preference Alignment

arXiv 2504.06659 ↗First posted 9 Apr 2025cs.LG cs.AI cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

Despite advances in Preference Alignment (PA) for Large Language Models (LLMs), mainstream methods like reinforcement learning with human feedback face notable challenges. These approaches require high-quality datasets of positive preference examples, which are costly to obtain and computationally intensive. The LLM unlearning technique presents a promising alternative by directly removing the influence of negative examples. However, current research has primarily focused on empirical validation, lacking systematic quantitative analysis. To bridge this gap, we propose a framework linking PA with LLM unlearning. Through bi-level optimization, we first quantify how unlearning specific negative examples impacts PA performance. Our analysis reveals that these effects vary substantially across negative examples. Building on this insight, we pose a crucial question: how can we optimally select and weight negative examples for unlearning to maximize PA performance? To answer this, we propose Unlearning to Align (U2A), which leverages bi-level optimization to efficiently select and unlearn examples for optimal PA performance. We validate the proposed method through extensive experiments, with results confirming its effectiveness. Our code is available at https://github.com/muyiahhh/U2A.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 69 of its 87 references that are in the map. First posted on arXiv 9 Apr 2025.

Builds on

69 of 87 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment19 Dec
  2. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning9 Oct
  3. 2024WPO: Enhancing RLHF with Weighted Preference Optimization17 Jun
  4. 2024Aligning Large Language Models with Representation Editing: A Control Perspective10 Jun
  5. 2024Transfer Q Star: Principled Decoding for LLM Alignment30 May
  6. 2024Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models29 May
  7. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  8. 2024DeAL: Decoding-time Alignment for Large Language Models5 Feb
  9. 2024Decoding-time Realignment of Language Models5 Feb
  10. 2024ARGS: Alignment as Reward-Guided Search23 Jan
  11. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
1 more
  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 56

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2024Qwen2 Technical Reportcited by 2,120
  10. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  11. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
38 more
  1. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  3. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  4. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 533
  5. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  6. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 437
  7. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 429
  8. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  9. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  10. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
  11. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  12. 2024Rethinking Machine Unlearning for Large Language Modelscited by 257
  13. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  14. 2016Hyperparameter optimization with approximate gradientcited by 233
  15. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  16. 2022A Contrastive Framework for Neural Text Generationcited by 225
  17. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 163
  18. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 159
  19. 2024Towards Safer Large Language Models through Machine Unlearningcited by 139
  20. 2023Controlled Decoding from Language Modelscited by 135
  21. 2022Continual Learning and Private Unlearningcited by 130
  22. 2023KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignmentcited by 99
  23. 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Differencecited by 88
  24. 2024LLM Unlearning via Loss Adjustment with Only Forget Datacited by 75
  25. 2024Uncovering Safety Risks of Large Language Models through Concept Activation Vectorcited by 67
  26. 2021Towards Gradient-based Bilevel Optimization with Non-convex Followers and Beyondcited by 63
  27. 2023An Emulator for Fine-Tuning Large Language Models using Small Language Modelscited by 61
  28. 2022Critic-Guided Decoding for Controlled Text Generationcited by 45
  29. 2021Sinkhorn Distributionally Robust Optimizationcited by 38
  30. 2023ReMasker: Imputing Tabular Data with Masked Autoencodingcited by 37
  31. 2021On the Convergence Theory for Hessian-Free Bilevel Algorithmscited by 35
  32. 2023STPrivacy: Spatio-Temporal Privacy-Preserving Action Recognitioncited by 29
  33. 2024To Forget or Not? Towards Practical Knowledge Unlearning for Large Language Modelscited by 26
  34. 2024Does Unlearning Truly Unlearn? A Black Box Evaluation of LLM Unlearning Methodscited by 26
  35. 2021Unified Group Fairness on Federated Learningcited by 15
  36. 2024Neglected Hessian component explains mysteries in Sharpness regularizationcited by 12
  37. 2024Taming Nonconvex Stochastic Mirror Descent with General Bregman Divergencecited by 11
  38. 2024First-Order Methods for Linearly Constrained Bilevel Optimizationcited by 10

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025DecepChain: Inducing Deceptive Reasoning in Large Language Modelscited by 3
  2. 2025The Realignment Problem: When Right becomes Wrong in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  2. 2026Unbiased Alignment for Large Language Models with Noisy Preferences
  3. 2023Large Language Model Unlearning
  4. 2024Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models
  5. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  6. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  7. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  8. 2025Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning