mlsanityData to 14 Oct 2026

Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding

arXiv 2506.07434 ↗First posted 9 Jun 2025cs.CL cs.AIcited by 7

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large Language Models (LLMs) require alignment with human preferences to avoid generating offensive, false, or meaningless content. Recently, low-resource methods for LLM alignment have been popular, while still facing challenges in obtaining both high-quality and aligned content. Motivated by the observation that the difficulty of generating aligned responses is concentrated at the beginning of decoding, we propose a novel framework, Weak-to-Strong Decoding (WSD), to enhance the alignment ability of base models by the guidance of a small aligned model. The small model first drafts well-aligned beginnings, followed by the large base model to continue the rest, controlled by a well-designed auto-switch mechanism. We also collect a new dataset, GenerAlign, to fine-tune a small-sized Pilot-3B as the draft model, which effectively enhances different base models under the WSD framework to outperform all baseline methods, while avoiding degradation on downstream tasks, termed as the alignment tax. Extensive experiments are further conducted to examine the impact of different settings and time efficiency, as well as analyses on the intrinsic mechanisms of WSD in depth.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 35 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

29 of 35 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization10 Oct
  2. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey4 Sep
  3. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignment24 Jun
  4. 2024Aligning Large Language Models with Representation Editing: A Control Perspective10 Jun
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment17 Mar
  7. 2024Aligner: Efficient Alignment by Learning to Correct4 Feb
  8. 2024ARGS: Alignment as Reward-Guided Search23 Jan
  9. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning4 Dec
  10. 2023RAIN: Your Language Models Can Align Themselves without Finetuning13 Sep
  11. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 18

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  10. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  12. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
6 more
  1. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  2. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  3. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  4. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  5. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 130
  6. 2023Won't Get Fooled Again: Answering Questions with False Premisescited by 48

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025CoSteer: Collaborative Decoding-Time Personalization via Local Delta Steeringcited by 13
  2. 2025Swarms of Large Language Model Agents for Protein Sequence Design with Experimental Validationcited by 5
  3. 2025A Survey on Training-free Alignment of Large Language Modelscited by 2
  4. 2025Mitigating Overthinking through Reasoning Shapingcited by 2
  5. 2026Inference-time Alignment via Sparse Junction Steeringcited by 2
  6. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  7. 2026A Coin Flip Per Token: Bernoulli Sparse Steering of Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DeAL: Decoding-time Alignment for Large Language Models
  2. 2024Your Weak LLM is Secretly a Strong Teacher for Alignment
  3. 2024Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model
  4. 2024Decoding-Time Language Model Alignment with Multiple Objectives
  5. 2024Improving Weak-to-Strong Generalization with Reliability-Aware Alignment
  6. 2026When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger
  7. 2025P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesis
  8. 2025Contrastive Weak-to-strong Generalization