mlsanityData to 7 Oct 2026

ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference

arXiv 2312.02554 ↗First posted 5 Dec 2023cs.LG cs.CLcited by 11

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning language models to human expectations, e.g., being helpful and harmless, has become a pressing challenge for large language models. A typical alignment procedure consists of supervised fine-tuning and preference learning. Most preference learning methods, such as RLHF and DPO, depend on pairwise preference data, which inadequately address scenarios where human feedback is point-wise, leading to potential information loss and suboptimal performance. Addressing this gap, we introduce Point-wise Direct Preference Optimization, a novel preference learning method designed to harness point-wise feedback effectively. Our work also uncovers a novel connection between supervised fine-tuning and point-wise preference learning, culminating in Unified Language Model Alignment, a single-step method that unifies the alignment with human demonstrations and point-wise preferences. Extensive experiments on point-wise preference datasets with binary or continuous labels validate the effectiveness of our methods. Our code and a new dataset with high-quality demonstration samples on harmlessness are released.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 38 references that are in the map. First posted on arXiv 5 Dec 2023.

Builds on

29 of 38 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Preference Ranking Optimization for Human Alignment30 Jun
  2. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignment4 Jun
  3. 2022Training language models to follow instructions with human feedback4 Mar
  4. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 25

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  12. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
13 more
  1. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  3. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  4. 2019ELI5: Long Form Question Answeringcited by 567
  5. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  6. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  7. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  8. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  9. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  10. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  11. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  12. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  13. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 3.

11 papers.

  1. 2025A Survey of Direct Preference Optimizationcited by 35
  2. 2025Diverse Preference Learning for Capabilities and Alignmentcited by 25
  3. 2024A Survey on Human Preference Learning for Large Language Modelscited by 23
  4. 2026Large Language Model Reasoning Failurescited by 20
  5. 2024PORT: Preference Optimization on Reasoning Tracescited by 6
  6. 2025Scalable Valuation of Human Feedback through Provably Robust Model Alignmentcited by 4
  7. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignmentcited by 4
  8. 2026Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gainscited by 2
  9. 2024Efficient Alignment of Large Language Models via Data Samplingcited by 1
  10. 2025Forewarned is Forearmed: Pre-Synthesizing Jailbreak-like Instructions to Enhance LLM Safety Guardrail to Potential Attackscited by 1
  11. 2026PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  2. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  3. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
  4. 2024Investigating on RLHF methodology
  5. 2023Aligning Language Models with Offline Learning from Human Feedback
  6. 2023On Diversified Preferences of Large Language Model Alignment
  7. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  8. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences