mlsanityData to 7 Oct 2026

Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization

arXiv 2608.09568 ↗First posted 10 Aug 2026cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) aggregates token-level log-probability ratios via uniform summation, implicitly treating all tokens as contributing equally to the preference signal. However, the contribution of individual tokens to the preference signal varies. We introduce token credit, which modulates each token's KL regularization based on its contribution to the preference outcome. We derive that effective token credit is proportional to the magnitude of each token's implicit reward, and observe that this quantity evolves substantially during training. This implies that static token credit becomes increasingly misaligned as training progresses. In this work, we propose Se-DPO (Self-Evolving Token Credit for DPO), a live mechanism that derives token credit from the model's own evolving internal signals during DPO training. Since the reward signal varies in reliability across positions, Se-DPO calibrates token credit based on both the strength and the confidence of each token's contribution. Se-DPO requires no external models, adding only a lightweight calibration network with minimal computational overhead. Experiments show that Se-DPO improves over DPO by up to 9.8 points on AlpacaEval~2 and 12.2 points on Arena-Hard.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 46 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

43 of 46 references.

Closest ancestors · 22

Same direction, newest first

  1. 2026Autoregressive Direct Preference Optimization10 Feb
  2. 2026TAB-PO: Preference Optimization with a Token-Level Adaptive Barrier for Token-Critical Structured Generation3 Feb
  3. 2026Small-Margin Preferences Still Matter-If You Train Them Right1 Feb
  4. 2025TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization17 Jun
  5. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization8 Jun
  6. 2025Token-Importance Guided Direct Preference Optimization26 May
  7. 2025Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimization24 May
  8. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation4 Mar
  9. 2025KL Penalty Control via Perturbation for Direct Preference Optimization18 Feb
  10. 2024SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Masks7 Oct
  11. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights6 Oct
  12. 2024Selective Preference Optimization via Token-Level Reward Function Estimation24 Aug
10 more
  1. 2024β-DPO: Direct Preference Optimization with Dynamic β11 Jul
  2. 2024WPO: Enhancing RLHF with Weighted Preference Optimization17 Jun
  3. 2024Bootstrapping Language Models with DPO Implicit Rewards14 Jun
  4. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback13 Jun
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  7. 2024Token-level Direct Preference Optimization18 Apr
  8. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  10. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  10. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  11. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  12. 2025Process Reinforcement through Implicit Rewardscited by 478
9 more
  1. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  2. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  3. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  4. 2024Explore the Reasoning Capability of LLMs in the Chess Testbedcited by 17
  5. 2025Discriminative Policy Optimization for Token-Level Reward Modelscited by 15
  6. 2024T-REG: Preference Optimization with Token-Level Reward Regularizationcited by 13
  7. 2026A-SR: Self-Evolving Agentic LLMs for Symbolic Regression via Hierarchical Coordinationcited by 4
  8. 2026RUBRIC: Realism--Utility Balanced Ranking for Imbalanced Classificationcited by 1
  9. 2026AutoSupervision: Closing the Feedback Loop in Scientific Workflows with Grounded Revision Verificationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective
  2. 2026LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization
  3. 2026Token-weighted Direct Preference Optimization with Attention
  4. 2026AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates
  5. 2026Masking Frequent Tokens Sharpens Direct Preference Optimization
  6. 2024Robust Preference Optimization through Reward Model Distillation
  7. 2026Boosting Direct Preference Optimization with Penalization
  8. 2026OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning