mlsanityData to 7 Oct 2026

DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models

arXiv 2503.04240 ↗First posted 6 Mar 2025cs.CLcited by 5

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Inference-time alignment provides an efficient alternative for aligning LLMs with humans. However, these approaches still face challenges, such as limited scalability due to policy-specific value functions and latency during the inference phase. In this paper, we propose a novel approach, Diffusion-styled Preference Optimization (model), which provides an efficient and policy-agnostic solution for aligning LLMs with humans. By directly performing alignment at sentence level, model~avoids the time latency associated with token-level generation. Designed as a plug-and-play module, model~can be seamlessly integrated with various base models to enhance their alignment. Extensive experiments on AlpacaEval 2, MT-bench, and HH-RLHF demonstrate that model~achieves superior alignment performance across various settings, achieving a favorable trade-off between alignment quality and inference-time latency. Furthermore, model~demonstrates model-agnostic scalability, significantly improving the performance of large models such as Llama-3-70B.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 73 of its 91 references that are in the map. First posted on arXiv 6 Mar 2025.

Builds on

73 of 91 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Transfer Q Star: Principled Decoding for LLM Alignment30 May
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024Value Augmented Sampling for Language Model Alignment and Personalization10 May
  4. 2024Token-level Direct Preference Optimization18 Apr
  5. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models25 Mar
  6. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  7. 2024DeAL: Decoding-time Alignment for Large Language Models5 Feb
  8. 2024Aligner: Efficient Alignment by Learning to Correct4 Feb
  9. 2024ARGS: Alignment as Reward-Guided Search23 Jan
  10. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Training7 Nov
  11. 2023Large Language Model Alignment: A Survey26 Sep
  12. 2023RAIN: Your Language Models Can Align Themselves without Finetuning13 Sep
2 more
  1. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuning1 Jun
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 59

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023Let's Verify Step by Stepcited by 3,598
  11. 2023Mistral 7Bcited by 3,065
  12. 2020Learning to summarize from human feedbackcited by 2,562
38 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2023Consistency Modelscited by 1,778
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  5. 2017Hindsight Experience Replaycited by 1,096
  6. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  7. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  8. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  11. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  12. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  13. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Modelscited by 439
  14. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  15. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  16. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Controlcited by 191
  17. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  18. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editingcited by 182
  19. 2023Likelihood-Based Diffusion Language Modelscited by 147
  20. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  21. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
  22. 2023Controlled Decoding from Language Modelscited by 133
  23. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  24. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123
  25. 2022Latent Diffusion for Language Generationcited by 110
  26. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answeringcited by 73
  27. 2024CLLMs: Consistency Large Language Modelscited by 71
  28. 2023Fast Model Debias with Machine Unlearningcited by 70
  29. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  30. 2023PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Modelcited by 56
  31. 2023See and Think: Embodied Agent in Virtual Environmentcited by 54
  32. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Modelscited by 45
  33. 2023DiffuSeq-v2: Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Modelscited by 36
  34. 2024FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMscited by 36
  35. 2023DiffFashion: Reference-based Fashion Design with Structure-aware Transfer by Diffusion Modelscited by 33
  36. 2024Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Modelscited by 26
  37. 2024Learnable Privacy Neurons Localization in Language Modelscited by 26
  38. 2024CityCraft: A Real Crafter for 3D City Generationcited by 25

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025A Survey on Diffusion Language Modelscited by 75
  2. 2025Datasets and Recipes for Video Temporal Grounding via Reinforcement Learningcited by 18
  3. 2025BiasGuard: A Reasoning-enhanced Bias Detection Tool For Large Language Modelscited by 4
  4. 2025Detection, Classification, and Mitigation of Gender Bias in Large Language Modelscited by 1
  5. 2026JAGG: Jacobian-Aggregated Group Gradient for Efficient GRPO Training of Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPO
  2. 2025ComPO: Preference Alignment via Comparison Oracles
  3. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
  4. 2024Inference time LLM alignment in single and multidomain preference spectrum
  5. 2026A Zeroth-Order Paradigm for LLM Preference Alignment
  6. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment
  7. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  8. 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization