# TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights

arXiv: https://arxiv.org/abs/2410.04350 · HTML: https://mlsanity.com/paper/2410.04350/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 6 Oct 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 27 of 27)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://mlsanity.com/paper/2404.01318/) (2024)
- [UltraFeedback: Boosting Language Models with Scaled AI Feedback](https://mlsanity.com/paper/2310.01377/) (2023)
- [Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs](https://mlsanity.com/paper/2406.18629/) (2024)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [From r to Q^*: Your Language Model is Secretly a Q-Function](https://mlsanity.com/paper/2404.12358/) (2024)
- [Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive](https://mlsanity.com/paper/2402.13228/) (2024)
- [Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning](https://mlsanity.com/paper/2405.00451/) (2024)
- [A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems](https://mlsanity.com/paper/2203.01387/) (2022)
- [Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data](https://mlsanity.com/paper/2404.14367/) (2024)
- [Token-level Direct Preference Optimization](https://mlsanity.com/paper/2404.11999/) (2024)
- [A Survey of Text Watermarking in the Era of Large Language Models](https://mlsanity.com/paper/2312.07913/) (2023)
- [Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective](https://mlsanity.com/paper/2404.04626/) (2024)
- [Learning Optimal Advantage from Preferences and Mistaking it for Reward](https://mlsanity.com/paper/2310.02456/) (2023)
- [Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation](https://mlsanity.com/paper/2402.11907/) (2024)

## Built on by (most cited first, 30 of 42)

- [A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future](https://mlsanity.com/paper/2504.12328/) (2025)
- [A Survey of Direct Preference Optimization](https://mlsanity.com/paper/2503.11701/) (2025)
- [Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMs](https://mlsanity.com/paper/2506.21656/) (2025)
- [Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generation](https://mlsanity.com/paper/2508.10858/) (2025)
- [Mark Your LLM: Detecting the Misuse of Open-Source Large Language Models via Watermarking](https://mlsanity.com/paper/2503.04636/) (2025)
- [Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoning](https://mlsanity.com/paper/2601.07408/) (2026)
- [Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models](https://mlsanity.com/paper/2508.07173/) (2025)
- [ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selection](https://mlsanity.com/paper/2601.09195/) (2026)
- [AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation](https://mlsanity.com/paper/2503.02832/) (2025)
- [A Survey on Progress in LLM Alignment from the Perspective of Reward Design](https://mlsanity.com/paper/2505.02666/) (2025)
- [Token-Importance Guided Direct Preference Optimization](https://mlsanity.com/paper/2505.19653/) (2025)
- [SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Masks](https://mlsanity.com/paper/2410.05102/) (2024)
- [AMPO: Active Multi-Preference Optimization for Self-play Preference Selection](https://mlsanity.com/paper/2502.18293/) (2025)
- [Sharpness-Guided Group Relative Policy Optimization via Probability Shaping](https://mlsanity.com/paper/2511.00066/) (2025)
- [Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees](https://mlsanity.com/paper/2502.12678/) (2025)
- [SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization](https://mlsanity.com/paper/2511.06222/) (2025)
- [ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignment](https://mlsanity.com/paper/2512.06196/) (2025)
- [CTPD: Cross Tokenizer Preference Distillation](https://mlsanity.com/paper/2601.11865/) (2026)
- [Ambiguity Awareness Optimization: Towards Semantic Disambiguation for Direct Preference Optimization](https://mlsanity.com/paper/2511.23391/) (2025)
- [PIPA: Preference Alignment as Prior-Informed Statistical Estimation](https://mlsanity.com/paper/2502.05773/) (2025)
- [Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms](https://mlsanity.com/paper/2506.09457/) (2025)
- [From Utterance to Vividity: Training Expressive Subtitle Translation LLM via Adaptive Local Preference Optimization](https://mlsanity.com/paper/2602.01068/) (2026)
- [Mitigating Mismatch within Reference-based Preference Optimization](https://mlsanity.com/paper/2602.11902/) (2026)
- [Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models](https://mlsanity.com/paper/2605.02626/) (2026)
- [Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelines](https://mlsanity.com/paper/2605.26442/) (2026)
- [SIPO: Stabilized and Improved Preference Optimization for Aligning Diffusion Models](https://mlsanity.com/paper/2505.21893/) (2025)
- [Preference Distillation via Value based Reinforcement Learning](https://mlsanity.com/paper/2509.16965/) (2025)
- [Data-efficient Targeted Token-level Preference Optimization for LLM-based Text-to-Speech](https://mlsanity.com/paper/2510.05799/) (2025)
- [Supervised Fine-Tuning Needs to Unlock the Potential of Token Priority](https://mlsanity.com/paper/2602.01227/) (2026)
- [Learning Evidence Highlighting for Frozen LLMs](https://mlsanity.com/paper/2604.22565/) (2026)

## Abstract (arXiv)

Direct Preference Optimization (DPO) has been widely adopted for preference alignment of Large Language Models (LLMs) due to its simplicity and effectiveness. However, DPO is derived as a bandit problem in which the whole response is treated as a single arm, ignoring the importance differences between tokens, which may affect optimization efficiency and make it difficult to achieve optimal results. In this work, we propose that the optimal data for DPO has equal expected rewards for each token in winning and losing responses, as there is no difference in token importance. However, since the optimal dataset is unavailable in practice, we propose using the original dataset for importance sampling to achieve unbiased optimization. Accordingly, we propose a token-level importance sampling DPO objective named TIS-DPO that assigns importance weights to each token based on its reward. Inspired by previous works, we estimate the token importance weights using the difference in prediction probabilities from a pair of contrastive LLMs. We explore three methods to construct these contrastive LLMs: (1) guiding the original LLM with contrastive prompts, (2) training two separate LLMs using winning and losing responses, and (3) performing forward and reverse DPO training with winning and losing responses. Experiments show that TIS-DPO significantly outperforms various baseline methods on harmlessness and helpfulness alignment and summarization tasks. We also visualize the estimated weights, demonstrating their ability to identify key token positions.