mlsanityData to 7 Oct 2026

Semi-Supervised Preference Optimization with Limited Feedback

arXiv 2511.00040 ↗First posted 28 Oct 2025cs.LG cs.AIcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The field of preference optimization has made outstanding contributions to the alignment of language models with human preferences. Despite these advancements, recent methods still rely heavily on substantial paired (labeled) feedback data, leading to substantial resource expenditures. To address these challenges, we study the problem of Semi-Supervised Preference Optimization (SSPO) in which the idea is to learn from both a small number of pairwise preference labels and a large pool of unpaired samples simultaneously. Our key theoretical contribution proves the existence of an optimal reward threshold capable of separating winning and losing responses with high probability, which enables a principled pseudo-labeling of unpaired data. By leveraging these pseudo-labels, SSPO effectively distills latent preferences from large-scale unpaired data, thus maintaining human alignment while drastically reducing acquisition costs. Extensive experiments across datasets validate this remarkable data efficiency; for instance, SSPO trained with Mistral-7B-Instruct on just 1% of UltraFeedback consistently surpasses strong baselines trained on 10% of UltraFeedback.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 47 references that are in the map. First posted on arXiv 28 Oct 2025.

Builds on

31 of 47 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation10 Oct
  2. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment6 Jun
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  4. 2024Human Alignment of Large Language Models through Online Preference Optimisation13 Mar
  5. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  6. 2023Aligning Large Language Models with Human Preferences through Representation Engineering26 Dec
  7. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 24

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
12 more
  1. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  2. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 507
  3. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  4. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  5. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  6. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 42
  7. 2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33
  8. 2024Small Language Models Learn Enhanced Reasoning Skills from Medical Textbookscited by 30
  9. 2023Safer-Instruct: Aligning Language Models with Automated Preference Datacited by 27
  10. 2024Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoningcited by 26
  11. 2024Semi-Supervised Reward Modeling via Iterative Self-Trainingcited by 8
  12. 2024Dr. SoW: Density Ratio of Strong-over-weak LLMs for Reducing the Cost of Human Annotation in Preference Tuningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 15
  2. 2026ABC-Align: Prediction-Powered Alignment with Adaptive Bias Controlcited by 0
  3. 2026Range-GRPO: Policy Optimization via Pairwise Relations among Reward Intervalscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
  2. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  3. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  4. 2024Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness
  5. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignment
  6. 2026Nearly Optimal Active Preference Learning and Its Application to LLM Alignment
  7. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO
  8. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier