mlsanityData to 7 Oct 2026

REFA: Reference Free Alignment for multi-preference optimization

arXiv 2412.16378 ↗First posted 20 Dec 2024cs.LG cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

To mitigate reward hacking from response verbosity, modern preference optimization methods are increasingly adopting length normalization (e.g., SimPO, ORPO, LN-DPO). While effective against this bias, we demonstrate that length normalization itself introduces a failure mode: the URSLA shortcut. Here models learn to satisfy the alignment objective by prematurely truncating low-quality responses rather than learning from their semantic content. To address this, we introduce REFA, a new alignment framework that proposes probabilistic control on a structural token that controls termination. Our core innovation is a new class of regularizers that operate directly on the probability of the End-of-Sequence (EOS) token, a previously unexploited control lever. This token-level intervention provides a principled solution to the URSLA shortcut, ensuring genuine quality improvements. Furthermore, it unlocks a versatile mechanism for managing the alignment-efficiency tradeoff, enabling practitioners to fine-tune models that adhere to specific token budgets. Empirically, REFA achieves a 60.29% win rate and a 52.17% length-controlled win rate on AlpacaEval2 with Llama-3-8B-Instruct, demonstrating the power of our token-level control paradigm.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 50 references that are in the map. First posted on arXiv 20 Dec 2024.

Builds on

38 of 50 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts5 Dec
  2. 2024Preference Optimization with Multi-Sample Comparisons16 Oct
  3. 2024A Practical Analysis of Human Alignment with *PO21 Jul
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  5. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  6. 2024Token-level Direct Preference Optimization18 Apr
  7. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  8. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  9. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  10. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment29 Feb
  11. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  12. 2024Noise Contrastive Alignment of Language Models with Explicit Rewards8 Feb
3 more
  1. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  10. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  11. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  12. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
11 more
  1. 2022Matryoshka Representation Learningcited by 268
  2. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  3. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 155
  4. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  5. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  6. 2023Is RLHF More Difficult than Standard RL?cited by 96
  7. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 94
  8. 2023MatFormer: Nested Transformer for Elastic Inferencecited by 79
  9. 2024Datasets for Large Language Models: A Comprehensive Surveycited by 69
  10. 2024Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Modelscited by 64
  11. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  2. 2025ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
  3. 2026SLIME: Stabilized Likelihood Implicit Margin Enforcement for Preference Optimization
  4. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO
  5. 2025Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Data
  6. 2026How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics
  7. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment
  8. 2026Mitigating Mismatch within Reference-based Preference Optimization