mlsanityData to 7 Oct 2026

SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling

arXiv 2405.12739 ↗First posted 21 May 2024cs.LGcited by 21

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Human preference alignment is critical in building powerful and reliable large language models (LLMs). However, current methods either ignore the multi-dimensionality of human preferences (e.g. helpfulness and harmlessness) or struggle with the complexity of managing multiple reward models. To address these issues, we propose Sequential Preference Optimization (SPO), a method that sequentially fine-tunes LLMs to align with multiple dimensions of human preferences. SPO avoids explicit reward modeling, directly optimizing the models to align with nuanced human preferences. We theoretically derive closed-form optimal SPO policy and loss function. Gradient analysis is conducted to show how SPO manages to fine-tune the LLMs while maintaining alignment on previously optimized dimensions. Empirical results on LLMs of different size and multiple evaluation datasets demonstrate that SPO successfully aligns LLMs across multiple dimensions of human preferences and significantly outperforms the baselines.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 51 references that are in the map. First posted on arXiv 21 May 2024.

Builds on

38 of 51 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  2. 2023Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game14 Nov
  3. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  4. 2022Training language models to follow instructions with human feedback4 Mar
  5. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023Mistral 7Bcited by 3,065
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
21 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  6. 2024Mixtral of Expertscited by 1,785
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  9. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  10. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  11. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  12. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  13. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  14. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  15. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  16. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  17. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  18. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  19. 2023Language Modeling Is Compressioncited by 225
  20. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  21. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 12; 2026: 4.

21 papers.

Sort:

By area:

  1. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
  2. 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modelingcited by 60
  3. 2024Personality Alignment of Large Language Modelscited by 36
  4. 2025A Survey of Direct Preference Optimizationcited by 35
  5. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  6. 2024Learning to Summarize from LLM-generated Feedbackcited by 24
  7. 2025Understanding and Mitigating Overrefusal in LLMs from an Unveiling Perspective of Safety Decision Boundarycited by 17
  8. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  9. 2025AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Modelscited by 6
  10. 2024Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimizationcited by 5
  11. 2025REWARD CONSISTENCY: Improving Multi-Objective Alignment from a Data-Centric Perspectivecited by 5
  12. 2026Beyond Perfect APIs: A Comprehensive Evaluation of LLM Agents Under Real-World API Complexitycited by 5
9 more
  1. 2025ReFeed: Multi-dimensional Summarization Refinement with Reflective Reasoning on Feedbackcited by 4
  2. 2025SynCast: Synergizing Contradictions in Precipitation Nowcasting via Diffusion Sequential Preference Optimizationcited by 2
  3. 2025Is On-Policy Data always the Best Choice for Direct Preference Optimization-based LM Alignment?cited by 1
  4. 2025Multi-dimensional Preference Alignment by Conditioning Reward Itselfcited by 1
  5. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  6. 2025Dual Caption Preference Optimization for Diffusion Modelscited by 0
  7. 2025References Indeed Matter? Reference-Free Preference Optimization for Conversational Query Reformulationcited by 0
  8. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Modelscited by 0
  9. 2026Toward a Theory of Value in AI Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025IPO: Your Language Model is Secretly a Preference Classifier
  2. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  3. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  4. 2025Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
  5. 2024A Survey on Human Preference Learning for Large Language Models
  6. 2024Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness
  7. 2024Transforming and Combining Rewards for Aligning Large Language Models
  8. 2025BPO: Revisiting Preference Modeling in Direct Preference Optimization