mlsanityData to 7 Oct 2026

Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

arXiv 2312.09390 ↗First posted 14 Dec 2023cs.CLcited by 445

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Widely used alignment techniques, such as reinforcement learning from human feedback (RLHF), rely on the ability of humans to supervise model behavior - for example, to evaluate whether a model faithfully followed instructions or generated safe outputs. However, future superhuman models will behave in complex ways too difficult for humans to reliably evaluate; humans will only be able to weakly supervise superhuman models. We study an analogy to this problem: can weak model supervision elicit the full capabilities of a much stronger model? We test this using a range of pretrained language models in the GPT-4 family on natural language processing (NLP), chess, and reward modeling tasks. We find that when we naively finetune strong pretrained models on labels generated by a weak model, they consistently perform better than their weak supervisors, a phenomenon we call weak-to-strong generalization. However, we are still far from recovering the full capabilities of strong models with naive finetuning alone, suggesting that techniques like RLHF may scale poorly to superhuman models without further work. We find that simple methods can often significantly improve weak-to-strong generalization: for example, when finetuning GPT-4 with a GPT-2-level supervisor and an auxiliary confidence loss, we can recover close to GPT-3.5-level performance on NLP tasks. Our results suggest that it is feasible to make empirical progress today on a fundamental challenge of aligning superhuman models.

Key facts. Cited by 445 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 107 of its 143 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

107 of 143 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Discovering Language Model Behaviors with Model-Written Evaluations19 Dec
  2. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  3. 2020Learning to summarize from human feedback2 Sep
  4. 2020Aligning AI With Shared Human Values5 Aug

Other references · 103

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  12. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
38 more
  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  13. 2020Knowledge Distillation: A Surveycited by 1,578
  14. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  15. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  16. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  17. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  18. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  19. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  20. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  21. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  22. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  23. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  24. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  25. 2022Red Teaming Language Models with Language Modelscited by 983
  26. 2018Neural Network Acceptability Judgmentscited by 952
  27. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  28. 2021Robust fine-tuning of zero-shot modelscited by 785
  29. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  30. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  31. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  32. 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 691
  33. 2018Mitigating Unwanted Biases with Adversarial Learningcited by 663
  34. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  35. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  36. 2018Born Again Neural Networkscited by 539
  37. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  38. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522

50 most cited of 103 All 103 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 184; 2025: 131; 2026: 128.

445 papers.

Sort:

By area:

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  3. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  4. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  5. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  6. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  7. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  8. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  9. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  10. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  11. 2025Learning to Reason without External Rewardscited by 199
  12. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
38 more
  1. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  2. 2025Less-to-More Generalization: Unlocking More Controllability by In-Context Generationcited by 153
  3. 2024Nemotron-4 340B Technical Reportcited by 143
  4. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  5. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 134
  6. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  7. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 105
  8. 2024Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignmentcited by 104
  9. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  10. 2024Improving Multi-Agent Debate with Sparse Communication Topologycited by 103
  11. 2024Aligner: Efficient Alignment by Learning to Correctcited by 96
  12. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  13. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 94
  14. 2024On scalable oversight with weak LLMs judging strong LLMscited by 94
  15. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 92
  16. 2024Tuning Language Models by Proxycited by 88
  17. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  18. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 80
  19. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  20. 2023Alleviating Hallucinations of Large Language Models through Induced Hallucinationscited by 78
  21. 2025Persona Features Control Emergent Misalignmentcited by 76
  22. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 69
  23. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  24. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 67
  25. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 60
  26. 2024Can Language Models Learn to Skip Steps?cited by 59
  27. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
  28. 2024Dynamic Evaluation of Large Language Models by Meta Probing Agentscited by 54
  29. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  30. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  31. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  32. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 51
  33. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
  34. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  35. 2025Chain-of-Retrieval Augmented Generationcited by 48
  36. 2024Enhancing LLM Safety via Constrained Direct Preference Optimizationcited by 47
  37. 2024Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulationcited by 46
  38. 2024Theoretical Analysis of Weak-to-Strong Generalizationcited by 46

50 most cited of 445 All 445 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Selective Weak-to-Strong Generalization
  2. 2024Reward Modeling with Weak Supervision for Language Models
  3. 2024Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learning
  4. 2024Strong and weak alignment of large language models with human values
  5. 2023SuperHF: Supervised Iterative Learning from Human Feedback
  6. 2024Self-Rewarding Language Models
  7. 2025Multiple Weaks Win Single Strong: Large Language Models Ensemble Weak Reinforcement Learning Agents into a Supreme One
  8. 2025Generalist Reward Models: Found Inside Large Language Models