# Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective

arXiv: https://arxiv.org/abs/2502.14340 · HTML: https://mlsanity.com/paper/2502.14340/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 2 of its 49 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 2 of 2)

- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)

## Built on by (most cited first, 11 of 11)

- [A Survey of Direct Preference Optimization](https://mlsanity.com/paper/2503.11701/) (2025)
- [Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMs](https://mlsanity.com/paper/2506.21656/) (2025)
- [Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2510.24302/) (2025)
- [Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs](https://mlsanity.com/paper/2506.10054/) (2025)
- [TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization](https://mlsanity.com/paper/2506.14574/) (2025)
- [Love First, Know Later: Persona-Based Romantic Compatibility Through LLM Text World Engines](https://mlsanity.com/paper/2512.11844/) (2025)
- [C2-DPO: Constrained Controlled Direct Preference Optimization](https://mlsanity.com/paper/2502.17507/) (2025)
- [Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms](https://mlsanity.com/paper/2506.09457/) (2025)
- [Autoregressive Direct Preference Optimization](https://mlsanity.com/paper/2602.09533/) (2026)
- [DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)](https://mlsanity.com/paper/2507.07855/) (2025)
- [Token-weighted Direct Preference Optimization with Attention](https://mlsanity.com/paper/2605.21883/) (2026)

## Abstract (arXiv)

Direct Preference Optimization (DPO) has gained attention as an efficient alternative to reinforcement learning from human feedback (RLHF) for aligning large language models (LLMs) with human preferences. Despite its advantages, DPO suffers from a length bias, generating responses longer than those from the reference model. Existing solutions like SimPO and SamPO address this issue but uniformly treat the contribution of rewards across sequences, overlooking temporal dynamics. To this end, we propose an enhanced preference optimization method that incorporates a temporal decay factor controlled by a gamma parameter. This dynamic weighting mechanism adjusts the influence of each reward based on its position in the sequence, prioritizing earlier tokens that are more critical for alignment. By adaptively focusing on more relevant feedback, our approach mitigates overfitting to less pertinent data and remains responsive to evolving human preferences. Experimental results on several benchmarks show that our approach consistently outperforms vanilla DPO by 5.9-8.8 points on AlpacaEval 2 and 3.3-9.7 points on Arena-Hard across different model architectures and sizes. Furthermore, additional experiments on mathematical and reasoning benchmarks (MMLU, GSM8K, and MATH) confirm that our method enhances performance without compromising general capabilities. Our codebase would be available at urlhttps://github.com/LotuSrc/D2PO.