mlsanityData to 7 Oct 2026

Your Weak LLM is Secretly a Strong Teacher for Alignment

arXiv 2409.08813 ↗First posted 13 Sep 2024cs.CLcited by 20

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The burgeoning capabilities of large language models (LLMs) have underscored the need for alignment to ensure these models act in accordance with human values and intentions. Existing alignment frameworks present constraints either in the form of expensive human effort or high computational costs. This paper explores a promising middle ground, where we employ a weak LLM that is significantly less resource-intensive than top-tier models, yet offers more automation than purely human feedback. We present a systematic study to evaluate and understand weak LLM's ability to generate feedback for alignment. Our empirical findings demonstrate that weak LLMs can provide feedback that rivals or even exceeds that of fully human-annotated data. Our study indicates a minimized impact of model size on feedback efficacy, shedding light on a scalable and sustainable alignment strategy. To deepen our understanding of alignment under weak LLM feedback, we conduct a series of qualitative and quantitative analyses, offering novel insights into the quality discrepancies between human feedback vs. weak LLM feedback.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 59 references that are in the map. First posted on arXiv 13 Sep 2024.

Builds on

45 of 59 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  2. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  3. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets20 Jul
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2023Mistral 7Bcited by 3,065
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
29 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  3. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  6. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  7. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  8. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  9. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  10. 2024Self-Rewarding Language Modelscited by 654
  11. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  12. 2018Born Again Neural Networkscited by 539
  13. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  14. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  15. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  16. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  17. 2022Is GPT-3 a Good Data Annotator?cited by 208
  18. 2018Progressive Feature Alignment for Unsupervised Domain Adaptationcited by 183
  19. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  20. 2023Self-Alignment with Instruction Backtranslationcited by 149
  21. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 134
  22. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  23. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  24. 2020Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predictcited by 75
  25. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
  26. 2024The Unreasonable Effectiveness of Easy Training Data for Hard Taskscited by 40
  27. 2024Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Expertscited by 28
  28. 2022Optimal Weak to Strong Learningcited by 14
  29. 2024A Moral Imperative: The Need for Continual Superalignment of Large Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 13; 2026: 5.

20 papers.

Sort:

By area:

  1. 2025Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executorscited by 20
  2. 2024Provable Weak-to-Strong Generalization via Benign Overfittingcited by 16
  3. 2025Weak-to-Strong Generalization Even in Random Feature Networks, Provablycited by 13
  4. 2025Incentivizing Strong Reasoning from Weak Supervisioncited by 8
  5. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combatcited by 8
  6. 2025On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspectivecited by 7
  7. 2025Unraveling Misinformation Propagation in LLM Reasoningcited by 6
  8. 2024Challenges and Future Directions of Data-Centric AI Alignmentcited by 5
  9. 2026Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversightcited by 5
  10. 2025Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Alignercited by 4
  11. 2025Super Co-alignment of Human and AI for Sustainable Symbiotic Societycited by 2
  12. 2025Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignmentcited by 2
8 more
  1. 2025Limited Preference Data? Learning Better Reward Model with Latent Space Synthesiscited by 2
  2. 2025General Exploratory Bonus for Optimistic Exploration in RLHFcited by 2
  3. 2025Contrastive Weak-to-strong Generalizationcited by 2
  4. 2025More Than the Final Answer: Improving Visual Extraction and Logical Consistency in Vision-Language Modelscited by 1
  5. 2026The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledgecited by 1
  6. 2026When Weak LLMs Speak with Confidence, Preference Alignment Gets Strongercited by 0
  7. 2026Trust Functions: Near-Lossless Weak-to-Strong Generalization by Learning When to Trust the Weak Teachercited by 0
  8. 2026ABC-Align: Prediction-Powered Alignment with Adaptive Bias Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  2. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning
  3. 2025A Comprehensive Evaluation framework of Alignment Techniques for LLMs
  4. 2025Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding
  5. 2025On Evaluating LLM Alignment by Evaluating LLMs as Judges
  6. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language
  7. 2023Large Language Model Alignment: A Survey
  8. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment