mlsanityData to 7 Oct 2026

Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models

arXiv 2405.19262 ↗First posted 29 May 2024cs.CL cs.AI cs.LGcited by 32

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language models are usually fine-tuned to align with human preferences. However, fine-tuning a large language model can be challenging. In this work, we introduce textitweak-to-strong search, framing the alignment of a large language model as a test-time greedy search to maximize the log-probability difference between small tuned and untuned models while sampling from the frozen large model. This method serves both as (1) a compute-efficient model up-scaling strategy that avoids directly tuning the large model and as (2) an instance of weak-to-strong generalization that enhances a strong model with weak test-time guidance. Empirically, we demonstrate the flexibility of weak-to-strong search across different tasks. In controlled-sentiment generation and summarization, we use tuned and untuned textttgpt2s to improve the alignment of large models without additional training. Crucially, in a more difficult instruction-following benchmark, AlpacaEval 2.0, we show that reusing off-the-shelf small models (e.g., textttzephyr-7b-beta and its untuned version) can improve the length-controlled win rates of both white-box and black-box large models against textttgpt-4-turbo (e.g., 34.4% rightarrow 37.9% for textttLlama-3-70B-Instruct and 16.0% rightarrow 20.1% for textttgpt-3.5-turbo-instruct), despite the small models' low win rates ≈ 10.0%.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 59 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

42 of 59 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  2. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  3. 2024DeAL: Decoding-time Alignment for Large Language Models5 Feb
  4. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 217 Nov
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  6. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  7. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2023Mistral 7Bcited by 3,065
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2020Learning to summarize from human feedbackcited by 2,562
23 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  4. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  9. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  10. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  11. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  12. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  13. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  14. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  15. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  16. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  17. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  18. 2023Controlled Decoding from Language Modelscited by 133
  19. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
  20. 2024Tuning Language Models by Proxycited by 88
  21. 2023An Emulator for Fine-Tuning Large Language Models using Small Language Modelscited by 61
  22. 2022Critic-Guided Decoding for Controlled Text Generationcited by 45
  23. 2024Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!cited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 16; 2026: 8.

32 papers.

Sort:

By area:

  1. 2025Process Reinforcement through Implicit Rewardscited by 478
  2. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  3. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  4. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  5. 2024Discovering Preference Optimization Algorithms with and for Large Language Modelscited by 38
  6. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  7. 2024Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalizationcited by 23
  8. 2024Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Modelcited by 22
  9. 2025Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executorscited by 20
  10. 2025Emergent Response Planning in LLMscited by 17
  11. 2025Self-Guided Process Reward Optimization with Redefined Step-wise Advantage for Process Reinforcement Learningcited by 10
  12. 2025From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignmentcited by 9
20 more
  1. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 8
  2. 2025Inference-time Alignment in Continuous Spacecited by 7
  3. 2026Weak-Driven Learning: How Weak Agents make Strong Agents Strongercited by 5
  4. 2025How to Mitigate Overfitting in Weak-to-strong Generalization?cited by 3
  5. 2025Watermarking Degrades Alignment in Language Models: Analysis and Mitigationcited by 3
  6. 2026How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMscited by 3
  7. 2024Breaking PEFT Limitations: Leveraging Weak-to-Strong Knowledge Transfer for Backdoor Attacks in LLMscited by 2
  8. 2025Leveraging Machine Unlearning for Cost-Efficient Preference Alignmentcited by 2
  9. 2025Collaborative Learning of On-Device Small Model and Cloud-Based Large Model: Advances and Future Directionscited by 2
  10. 2025Contrastive Weak-to-strong Generalizationcited by 2
  11. 2026Inference-time Alignment via Sparse Junction Steeringcited by 2
  12. 2026Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillationcited by 2
  13. 2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approachcited by 1
  14. 2025Limitations of refinement methods for weak to strong generalizationcited by 1
  15. 2025Alignment-Aware Decodingcited by 1
  16. 2025Mitigating Object Hallucination via Robust Local Perception Searchcited by 0
  17. 2026When Weak LLMs Speak with Confidence, Preference Alignment Gets Strongercited by 0
  18. 2026Gradient-Guided Reward Optimization for Inference-time Alignmentcited by 0
  19. 2026Test-Time Weak-to-Strong Alignment: Transferring Implicit Rewards from Weak to Strong Flow Modelscited by 0
  20. 2026Allspark: Weak to Strong Transfer via Alternating Chain of Thoughtcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Improving Weak-to-Strong Generalization with Reliability-Aware Alignment
  2. 2024Bayesian WeakS-to-Strong from Text Classification to Generation
  3. 2025Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding
  4. 2024Inference-Time Language Model Alignment via Integrated Value Guidance
  5. 2024A transfer learning framework for weak-to-strong generalization
  6. 2025W2S-AlignTree: Weak-to-Strong Inference-Time Alignment for Large Language Models via Monte Carlo Tree Search
  7. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Models
  8. 2023Text Alignment Is An Efficient Unified Model for Massive NLP Tasks