mlsanityData to 14 Oct 2026

When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger

arXiv 2603.04968 ↗First posted 5 Mar 2026cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference alignment is an essential step in adapting large language models (LLMs) to human values, but existing approaches typically depend on costly human annotations or large-scale API-based models. We explore whether a weak LLM can instead act as an effective annotator. We surprisingly find that selecting only a subset of a weak LLM's highly confident samples leads to substantially better performance than using full human annotations. Building on this insight, we propose Confidence-Weighted Preference Optimization (CW-PO), a general framework that re-weights training samples by a weak LLM's confidence and can be applied across different preference optimization objectives. Notably, the model aligned by CW-PO with just 20% of human annotations outperforms the model trained with 100% of annotations under standard DPO. These results suggest that weak LLMs, when paired with confidence weighting, can dramatically reduce the cost of preference alignment while even outperforming methods trained on fully human-labeled data.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 34 references that are in the map. First posted on arXiv 5 Mar 2026.

Builds on

31 of 34 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model24 Oct
  2. 2024Your Weak LLM is Secretly a Strong Teacher for Alignment13 Sep
  3. 2024β-DPO: Direct Preference Optimization with Dynamic β11 Jul
  4. 2024WPO: Enhancing RLHF with Weighted Preference Optimization17 Jun
  5. 2024Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models29 May
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  7. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models15 Apr
  8. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  9. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  10. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  11. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  12. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
3 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  2. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets20 Jul
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 16

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  8. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  9. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  10. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  11. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  12. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
4 more
  1. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  2. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  3. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 106
  4. 2025Weak-to-Strong Generalization under Distribution Shiftscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
  2. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shifts
  3. 2024Dissecting Human and LLM Preferences
  4. 2025Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment
  5. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  6. 2026Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation
  7. 2024REAL: Response Embedding-based Alignment for LLMs
  8. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences