mlsanityData to 7 Oct 2026

Token-Importance Guided Direct Preference Optimization

arXiv 2505.19653 ↗First posted 26 May 2025cs.AIcited by 14

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning Large Language Models (LLMs) with human preferences is crucial for safe and effective AI interactions. While popular methods like Direct Preference Optimization (DPO) have simplified alignment, they remain sensitive to data noise and overlook the differential importance of individual tokens. Existing token-level approaches often rely on probability prediction or simplistic weighting schemes to obtain token importance, which still cannot fully address these issues. To solve this problem, we propose the Token-Importance Guided Direct Preference Optimization (TI-DPO), a framework that achieves fine-grained semantic control through two synergistic innovations. First, we propose a novel hybrid weighting mechanism that combines gradient attribution with a Gaussian prior, ensuring both the accuracy and robustness of token importance scores. Second, we employ a triplet loss to provide structured guidance for the optimization, explicitly guiding model outputs to approach preferred responses and diverge from non-preferred ones. Experimental results show that TI-DPO achieves higher accuracy and stronger generative diversity, providing more stable and computationally efficient solutions compared with DPO and other RLHF methods.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 49 of its 63 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

49 of 63 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Sequence-level Large Language Model Training with Contrastive Preference Optimization23 Feb
  2. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights6 Oct
  3. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment6 Jun
  4. 2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimization26 May
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  7. 2024Token-level Direct Preference Optimization18 Apr
  8. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model28 Mar
  9. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  10. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models15 Feb
  11. 2023Aligning Large Language Models with Human Preferences through Representation Engineering26 Dec
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
2 more
  1. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2017Deep reinforcement learning from human preferencescited by 3,846
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
23 more
  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  5. 2025Process Reinforcement through Implicit Rewardscited by 478
  6. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  7. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  8. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 260
  9. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  10. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  11. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  12. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  13. 2023A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directionscited by 101
  14. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 80
  15. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  16. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  17. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learningcited by 63
  18. 2024Preference Optimization for Reasoning with Pseudo Feedbackcited by 38
  19. 2024Distinguishing LLM-generated from Human-written Code by Contrastive Learningcited by 16
  20. 2024MixEval-X: Any-to-Any Evaluations from Real-World Data Mixturescited by 6
  21. 2025Sentence-level Reward Model can Generalize Better for Aligning LLM from Human Preferencecited by 5
  22. 2024Efficient Knowledge Distillation: Empowering Small Language Models with Teacher Model Insightscited by 4
  23. 2019Deep Reinforcement Learning with Feedback-based Explorationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 13.

14 papers.

Sort:

By area:

  1. 2026Chain-of-Thought Compression Should Not Be Blind: V-Skip for Efficient Multimodal Reasoning via Dual-Path Anchoringcited by 21
  2. 2026Not All Queries Need Deep Thought: CoFiCot for Adaptive Coarse-to-fine Stateful Refinementcited by 13
  3. 2025Not All Errors Are Created Equal: ASCoT Addresses Late-Stage Fragility in Efficient LLM Reasoningcited by 11
  4. 2026Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillationcited by 11
  5. 2026TAB-PO: Preference Optimization with a Token-Level Adaptive Barrier for Token-Critical Structured Generationcited by 5
  6. 2026Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarkscited by 1
  7. 2026Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winnercited by 1
  8. 2026Distribution Corrected Offline Data Distillation for Large Language Modelscited by 0
  9. 2026PVminerLLM2: Improving Structured Extraction of Patient Voice via Preference Optimizationcited by 0
  10. 2026KARMA: Knowledge graph-based Automated Reasoning Materialization and Alignmentcited by 0
  11. 2026Adaptive Model Compression (AMC): Saliency-Driven Resource Allocation for Ultra-Low-Power Transformer Inferencecited by 0
  12. 2026Se-DPO: Self-Evolving Token Credit for Direct Preference Optimizationcited by 0
2 more
  1. 2026Training LLM Judges from Language Feedback via Position-Selective Self-Distillationcited by 0
  2. 2026GAW-PO: Preference Optimization with Gradient-Aligned Token Weightscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Token-weighted Direct Preference Optimization with Attention
  2. 2025Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimization
  3. 2025Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective
  4. 2026TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
  5. 2025ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
  6. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  7. 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
  8. 2025A Survey of Direct Preference Optimization