mlsanityData to 14 Oct 2026

Self-supervised Attribute-aware Dynamic Preference Ranking Alignment

arXiv 2502.12189 ↗First posted 15 Feb 2025cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning from Human Feedback and its variants excel in aligning with human intentions to generate helpful, harmless, and honest responses. However, most of them rely on costly human-annotated pairwise comparisons for supervised alignment, which is not suitable for list-level scenarios, such as community question answering. Additionally, human preferences are influenced by multiple intrinsic factors in responses, leading to decision-making inconsistencies. Therefore, we propose Self-supervised Attribute-aware dynamic preference ranking, called shortname. It quantifies preference differences between responses based on Attribute-Perceptual Distance Factors (APDF) and dynamically determines the list-wise alignment order. Furthermore, it achieves fine-grained preference difference learning and enables precise alignment with the optimal one. We specifically constructed a challenging code preference dataset named StaCoCoQA, and introduced more cost-effective and scalable preference evaluation metrics: PrefHit and PrefRecall. Extensive experimental results show that SeAdpra exhibits superior performance and generalizability on both StaCoCoQA and preference datasets from eight popular domains.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 60 references that are in the map. First posted on arXiv 15 Feb 2025.

Builds on

27 of 60 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024LIRE: listwise reward enhancement for preference alignment22 May
  2. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences12 Mar
  3. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  4. 2024LiPO: Listwise Preference Optimization through Learning-to-Rank2 Feb
  5. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  6. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  7. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  8. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  5. 2024Qwen2 Technical Reportcited by 2,120
  6. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  10. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  11. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  12. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
7 more
  1. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  2. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  3. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 112
  4. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 102
  5. 2024Learning or Self-aligning? Rethinking Instruction Fine-tuningcited by 43
  6. 2024Enhancing Dialogue State Tracking Models through LLM-backed User-Agents Simulationcited by 24
  7. 2024HOLMES: Hyper-Relational Knowledge Graphs for Multi-hop Question Answering using LLMscited by 24

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Semi-Supervised Preference Optimization with Limited Feedback
  2. 2024Holistic Utility Preference Learning for Listwise Alignment
  3. 2024Permutative Preference Alignment from Listwise Ranking of Human Judgments
  4. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignment
  5. 2025Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors
  6. 2025Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
  7. 2026Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning
  8. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences