mlsanityData to 7 Oct 2026

Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model

arXiv 2410.18640 ↗First posted 24 Oct 2024cs.CLcited by 22

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning language models (LMs) with human preferences has become a key area of research, enabling these models to meet diverse user needs better. Inspired by weak-to-strong generalization, where a strong LM fine-tuned on labels generated by a weaker model can consistently outperform its weak supervisor, we extend this idea to model alignment. In this work, we observe that the alignment behavior in weaker models can be effectively transferred to stronger models and even exhibit an amplification effect. Based on this insight, we propose a method called Weak-to-Strong Preference Optimization (WSPO), which achieves strong model alignment by learning the distribution differences before and after the alignment of the weak model. Experiments demonstrate that WSPO delivers outstanding performance, improving the win rate of Qwen2-7B-Instruct on Arena-Hard from 39.70 to 49.60, achieving a remarkable 47.04 length-controlled win rate on AlpacaEval 2, and scoring 7.33 on MT-bench. Our results suggest that using the weak model to elicit a strong model with a high alignment ability is feasible.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 47 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

36 of 47 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Decoding-Time Language Model Alignment with Multiple Objectives27 Jun
  2. 2024Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models29 May
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  4. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  5. 2024Decoding-time Realignment of Language Models5 Feb
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct

Other references · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2020Learning to summarize from human feedbackcited by 2,562
18 more
  1. 2024Qwen2 Technical Reportcited by 2,076
  2. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  3. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  5. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  7. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  8. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  9. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  10. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  11. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  12. 2019A Theory of Regularized Markov Decision Processescited by 279
  13. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 155
  14. 2024Tuning Language Models by Proxycited by 88
  15. 2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75
  16. 2023An Emulator for Fine-Tuning Large Language Models using Small Language Modelscited by 61
  17. 2024Improving Open-Ended Text Generation via Adaptive Decodingcited by 21
  18. 2023CLEAN-EVAL: Clean Evaluation on Contaminated Large Language Modelscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 10.

22 papers.

Sort:

By area:

  1. 2025The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gainscited by 25
  2. 2025Proximal Supervised Fine-Tuningcited by 15
  3. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 5
  4. 2026Weak-Driven Learning: How Weak Agents make Strong Agents Strongercited by 5
  5. 2025W2S-AlignTree: Weak-to-Strong Inference-Time Alignment for Large Language Models via Monte Carlo Tree Searchcited by 4
  6. 2025Flexible Realignment of Language Modelscited by 3
  7. 2026Lookahead Sample Reward Guidance for Test-Time Scaling of Diffusion Modelscited by 3
  8. 2025The Capabilities and Limitations of Weak-to-Strong Generalization: Generalization and Calibrationcited by 2
  9. 2025Super Co-alignment of Human and AI for Sustainable Symbiotic Societycited by 2
  10. 2025Contrastive Weak-to-strong Generalizationcited by 2
  11. 2026An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shiftcited by 2
  12. 2026Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillationcited by 2
10 more
  1. 2025Limitations of refinement methods for weak to strong generalizationcited by 1
  2. 2025Alignment-Aware Decodingcited by 1
  3. 2026From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Mergingcited by 1
  4. 2025Adding Alignment Control to Language Modelscited by 0
  5. 2025BIRD: Behavior Induction via Representation-structure Distillationcited by 0
  6. 2026When Weak LLMs Speak with Confidence, Preference Alignment Gets Strongercited by 0
  7. 2026Maximizing Mutual Information Between Prompt and Response Improves LLM Performance With No Additional Datacited by 0
  8. 2026When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shiftcited by 0
  9. 2026AdaStop: Cost-Aware Early Stopping for DNN Test Selectioncited by 0
  10. 2026Trust Your Guide Only When Certain: Uncertainty-Aware Sparse Alignment at Inference Timecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding
  2. 2024Bayesian WeakS-to-Strong from Text Classification to Generation
  3. 2024MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization
  4. 2025Synergistic Weak-Strong Collaboration by Aligning Preferences
  5. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  6. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  7. 2023On Diversified Preferences of Large Language Model Alignment
  8. 2025Semi-Supervised Preference Optimization with Limited Feedback