mlsanityData to 14 Oct 2026

Token-weighted Direct Preference Optimization with Attention

arXiv 2605.21883 ↗First posted 21 May 2026cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) aligns Large Language Models with human preferences without the need for a separate reward model. However, DPO treats all tokens in responses equally, neglecting the differing importance of individual tokens. Existing token-level PO methods compute the token weights using either token-position-based heuristic functions or probability estimates given by a separately trained model, which lacks robustness and incurs extra training cost. In contrast, we propose Token-weighted DPO (TwDPO) -- a novel training objective grounded on token-weighted RL -- and AttentionPO -- an instantiation of TwDPO that uses attention from the LLM itself to estimate token weights. AttentionPO prompts the LLM to serve as a pairwise judge and check where the model attends when comparing the responses. This design makes AttentionPO content-aware, adjusting weights based on response content, and efficient, incurring only two extra forward passes per example. Experiment results show that AttentionPO significantly improves performance on AlpacaEval, MT-Bench, and ArenaHard, surpassing existing Preference Optimization methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 41 references that are in the map. First posted on arXiv 21 May 2026.

Builds on

30 of 41 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025DCRM: A Heuristic to Measure Response Pair Quality in Preference Optimization17 Jun
  2. 2025Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimization24 May
  3. 2025Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective20 Feb
  4. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights6 Oct
  5. 2024Selective Preference Optimization via Token-Level Reward Function Estimation24 Aug
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  7. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  8. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  10. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  8. 2019Attention is not Explanationcited by 929
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  10. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  12. 2020Quantifying Attention Flow in Transformerscited by 775
8 more
  1. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  2. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  3. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  4. 2019Is Attention Interpretable?cited by 389
  5. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  6. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 83
  7. 2019Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddingscited by 56
  8. 2025Attention-Based Reward Shaping for Sparse and Delayed Rewardscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Token-Importance Guided Direct Preference Optimization
  2. 2024Token-level Direct Preference Optimization
  3. 2026GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
  4. 2025A Survey of Direct Preference Optimization
  5. 2024Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts
  6. 2024Direct Preference Optimization with an Offset
  7. 2025FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference Rankings
  8. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts