mlsanityData to 7 Oct 2026

Autoregressive Direct Preference Optimization

arXiv 2602.09533 ↗First posted 10 Feb 2026cs.AIcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct preference optimization (DPO) has emerged as a promising approach for aligning large language models (LLMs) with human preferences. However, the widespread reliance on the response-level Bradley-Terry (BT) model may limit its full potential, as the reference and learnable models are assumed to be autoregressive only after deriving the objective function. Motivated by this limitation, we revisit the theoretical foundations of DPO and propose a novel formulation that explicitly introduces the autoregressive assumption prior to applying the BT model. By reformulating and extending DPO, we derive a novel variant, termed Autoregressive DPO (ADPO), that explicitly integrates autoregressive modeling into the preference optimization framework. Without violating the theoretical foundations, the derived loss takes an elegant form: it shifts the summation operation in the DPO objective outside the log-sigmoid function. Furthermore, through theoretical analysis of ADPO, we show that there exist two length measures to be considered when designing DPO-based algorithms: the token length μ and the feedback length μ'. To the best of our knowledge, we are the first to explicitly distinguish these two measures and analyze their implications for preference optimization in LLMs.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 49 of its 59 references that are in the map. First posted on arXiv 10 Feb 2026.

Builds on

49 of 59 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization17 Jun
  2. 2025ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization10 Jun
  3. 2025Explicit Preference Optimization: No Need for an Implicit Reward Model9 Jun
  4. 2025AMPO: Active Multi-Preference Optimization for Self-play Preference Selection25 Feb
  5. 2025Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective20 Feb
  6. 2025LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization19 Feb
  7. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback22 Jan
  8. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment19 Dec
  9. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization14 Oct
  10. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift26 Jul
  11. 2024β-DPO: Direct Preference Optimization with Dynamic β11 Jul
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
5 more
  1. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  2. 2024Token-level Direct Preference Optimization18 Apr
  3. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Models17 Oct
  4. 2023Aligning Language Models with Preferences through f-divergence Minimization16 Feb
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2025Qwen3 Technical Reportcited by 8,195
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2015Trust Region Policy Optimizationcited by 2,503
20 more
  1. 2025Gemma 3 Technical Reportcited by 1,686
  2. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  5. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  6. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  7. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  8. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  9. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  10. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  11. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  12. 2024Iterative Reasoning Preference Optimizationcited by 244
  13. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 94
  14. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  15. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 80
  16. 2024On Softmax Direct Preference Optimization for Recommendationcited by 64
  17. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  18. 2024Preference Optimization for Reasoning with Pseudo Feedbackcited by 38
  19. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Modelscited by 21
  20. 2023Privately Aligning Language Models with Reinforcement Learningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Se-DPO: Self-Evolving Token Credit for Direct Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Direct Preference Optimization with an Offset
  2. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  3. 2025A Survey of Direct Preference Optimization
  4. 2025What Matters in Data for DPO?
  5. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective
  6. 2024MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples
  7. 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge
  8. 2024Multi-Reference Preference Optimization for Large Language Models