mlsanityData to 14 Oct 2026

Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization

arXiv 2507.07725 ↗First posted 10 Jul 2025cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Post-training alignment of large language models (LLMs) is a critical challenge, as not all tokens contribute equally to model performance. This paper introduces a selective alignment strategy that prioritizes high-impact tokens within preference pairs, leveraging token-level log-probability differences between the current policy and a reference model. By focusing on these informative tokens, our approach reduces computational overhead and enhances alignment fidelity. We further explore the role of reference model quality, demonstrating that stronger reference models significantly improve token selection accuracy and overall optimization effectiveness. Comprehensive experiments on benchmarks such as Arena-Hard and MT-Bench validate the superiority of our Selective-DPO method over standard DPO and distillation-based baselines. Our findings highlight the importance of token-level optimization and reference model selection in advancing preference alignment for LLMs. The code is available at https://github.com/Dongzhijin/SDPO.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 10 of its 11 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

10 of 11 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Selective Preference Optimization via Token-Level Reward Function Estimation24 Aug
  2. 2024Direct Preference Knowledge Distillation for Large Language Models28 Jun
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May

Other references · 7

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  4. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  5. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  6. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  7. 2024Rho-1: Not All Tokens Are What You Needcited by 133

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  2. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  3. 2025ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
  4. 2026A Zeroth-Order Paradigm for LLM Preference Alignment
  5. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  6. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  7. 2025ComPO: Preference Alignment via Comparison Oracles
  8. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization