mlsanityData to 7 Oct 2026

A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques

arXiv 2406.04879 ↗First posted 7 Jun 2024cs.CLcited by 16

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language models are first pre-trained on trillions of tokens and then instruction-tuned or aligned to specific preferences. While pre-training remains out of reach for most researchers due to the compute required, fine-tuning has become affordable thanks to parameter-efficient methods such as LoRA and QLoRA. Alignment is known to be sensitive to the many factors involved, including the quantity and quality of data, the alignment method, and the adapter rank. However, there has not yet been an extensive study of their effect on downstream performance. To address this gap, we conduct an in-depth investigation of the impact of popular choices for three crucial axes: (i) the alignment dataset (HH-RLHF and BeaverTails), (ii) the alignment technique (SFT and DPO), and (iii) the model (LLaMA-1, Vicuna-v1.3, Mistral-7b, and Mistral-7b-Instruct). Our extensive setup spanning over 300 experiments reveals consistent trends and unexpected findings. We observe how more informative data helps with preference alignment, cases where supervised fine-tuning outperforms preference optimization, and how aligning to a distinct preference boosts performance on downstream tasks. Through our in-depth analyses, we put forward key guidelines to help researchers perform more effective parameter-efficient LLM alignment.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 26 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

19 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Parameter-Efficient Tuning Helps Language Model Alignment1 Oct

Other references · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
6 more
  1. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  3. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  4. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  5. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  6. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 10; 2026: 2.

16 papers.

Sort:

By area:

  1. 2025STAR-1: Safer Alignment of Reasoning LLMs with 1K Datacited by 67
  2. 2025A Survey of Direct Preference Optimizationcited by 35
  3. 2024Superficial Safety Alignment Hypothesiscited by 14
  4. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  5. 2025Safety Alignment Can Be Not Superficial With Explicit Safety Signalscited by 9
  6. 2025Value Drifts: Tracing Value Alignment During LLM Post-Trainingcited by 9
  7. 2024Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inferencecited by 7
  8. 2024Combining Domain and Alignment Vectors to Achieve Better Knowledge-Safety Trade-offs in LLMscited by 6
  9. 2025Unveiling Trust in Multimodal Large Language Models: Evaluation, Analysis, and Mitigationcited by 4
  10. 2025Enhancing Conversational Agents with Theory of Mind: Aligning Beliefs, Desires, and Intentions for Human-Like Interactioncited by 3
  11. 2025CAPO: Confidence Aware Preference Optimization Learning for Multilingual Preferencescited by 3
  12. 2025The Effectiveness of Approximate Regularized Replay for Efficient Supervised Fine-Tuning of Large Language Modelscited by 2
4 more
  1. 2024Anchored Alignment for Self-Explanations Enhancementcited by 1
  2. 2025Elementary Math Word Problem Generation using Large Language Modelscited by 0
  3. 2026The Hrunting of AI: Where and How to Improve English Dialectal Fairnesscited by 0
  4. 2026Fine-Tuning A Large Language Model for Systematic Review Screeningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  2. 2025ComPO: Preference Alignment via Comparison Oracles
  3. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  4. 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
  5. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters
  6. 2026A Zeroth-Order Paradigm for LLM Preference Alignment
  7. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  8. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignment