mlsanityData to 14 Oct 2026

Token-level Direct Preference Optimization

arXiv 2404.11999 ↗First posted 18 Apr 2024cs.CL cs.AIcited by 135

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Fine-tuning pre-trained Large Language Models (LLMs) is essential to align them with human values and intentions. This process often utilizes methods like pairwise comparisons and KL divergence against a reference LLM, focusing on the evaluation of full answers generated by the models. However, the generation of these responses occurs in a token level, following a sequential, auto-regressive fashion. In this paper, we introduce Token-level Direct Preference Optimization (TDPO), a novel approach to align LLMs with human preferences by optimizing policy at the token level. Unlike previous methods, which face challenges in divergence efficiency, TDPO incorporates forward KL divergence constraints for each token, improving alignment and diversity. Utilizing the Bradley-Terry model for a token-based reward system, TDPO enhances the regulation of KL divergence, while preserving simplicity without the need for explicit reward modeling. Experimental results across various text tasks demonstrate TDPO's superior performance in balancing alignment with generation diversity. Notably, fine-tuning with TDPO strikes a better balance than DPO in the controlled sentiment generation and single-turn dialogue datasets, and significantly improves the quality of generated responses compared to both DPO and PPO-based RLHF methods. Our code is open-sourced at https://github.com/Vance0124/Token-level-Direct-Preference-Optimization.

Key facts. Cited by 135 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 45 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

31 of 45 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  2. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  3. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  4. 2023Preference Ranking Optimization for Human Alignment30 Jun
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  11. 2015Trust Region Policy Optimizationcited by 2,580
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
14 more
  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  3. 2021Ethical and social risks of harm from Language Modelscited by 966
  4. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  6. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  7. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  8. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  9. 2020A Distributional Approach to Controlled Text Generationcited by 119
  10. 2022On Decoding Strategies for Neural Text Generatorscited by 90
  11. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  12. 2022Models of human preference for learning reward functionscited by 54
  13. 2022Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Modelscited by 49
  14. 2023Learning Optimal Advantage from Preferences and Mistaking it for Rewardcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 30; 2025: 65; 2026: 40.

135 papers.

Sort:

By area:

  1. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  3. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  4. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 96
  5. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 95
  6. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 84
  7. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 83
  8. 2024Process Reward Model with Q-Value Rankingscited by 80
  9. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  10. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  11. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  12. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
38 more
  1. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 51
  2. 2025Speculative Thinking: Enhancing Small-Model Reasoning with Large Model Guidance at Inference Timecited by 48
  3. 2025Baichuan-M1: Pushing the Medical Capability of Large Language Modelscited by 43
  4. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weightscited by 42
  5. 2025A Survey of Direct Preference Optimizationcited by 36
  6. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 32
  7. 2025A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuningcited by 30
  8. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 29
  9. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  10. 2025CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignmentcited by 26
  11. 2024Selective Preference Optimization via Token-Level Reward Function Estimationcited by 25
  12. 2025A Tutorial on LLM Reasoning: Relevant Methods behind ChatGPT o1cited by 25
  13. 2025Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMscited by 24
  14. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
  15. 2025GRPO-λ: Credit Assignment improves LLM Reasoningcited by 23
  16. 2025Understanding and Enhancing the Transferability of Jailbreaking Attackscited by 22
  17. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 21
  18. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 20
  19. 2024Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignmentcited by 20
  20. 2025VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Modelscited by 17
  21. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  22. 2025Token-Importance Guided Direct Preference Optimizationcited by 14
  23. 2025Learning Safety Constraints for Large Language Modelscited by 13
  24. 2024A Critical Look At Tokenwise Reward-Guided Text Generationcited by 12
  25. 2024Constrain Alignment with Sparse Autoencoderscited by 12
  26. 2025Trust Region Preference Approximation: A simple and stable reinforcement learning algorithm for LLM reasoningcited by 12
  27. 2024Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimizationcited by 11
  28. 2025Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHFcited by 11
  29. 2025VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detectioncited by 11
  30. 2025InSPO: Unlocking Intrinsic Self-Reflection for LLM Preference Optimizationcited by 11
  31. 2025ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimizationcited by 10
  32. 2024SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Maskscited by 9
  33. 2024Multi-Preference Optimization: Generalizing DPO via Set-Level Contrastscited by 9
  34. 2025Preference Optimization by Estimating the Ratio of the Data Distributioncited by 9
  35. 2025Decoupling Task-Solving and Output Formatting in LLM Generationcited by 9
  36. 2025FeRG-LLM : Feature Engineering by Reason Generation Large Language Modelscited by 8
  37. 2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Lawcited by 7
  38. 2024Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimizationcited by 7

50 most cited of 135 All 135 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Token-weighted Direct Preference Optimization with Attention
  2. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts
  3. 2025Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective
  4. 2026Tangent Space Fine-Tuning for Directional Preference Alignment in Large Language Models
  5. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
  6. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks
  7. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  8. 2024Multi-Reference Preference Optimization for Large Language Models