mlsanityData to 7 Oct 2026

Weak-to-Strong Reasoning

arXiv 2407.13647 ↗First posted 18 Jul 2024cs.CL cs.AIcited by 32

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

When large language models (LLMs) exceed human-level capabilities, it becomes increasingly challenging to provide full-scale and accurate supervision for these models. Weak-to-strong learning, which leverages a less capable model to unlock the latent abilities of a stronger model, proves valuable in this context. Yet, the efficacy of this approach for complex reasoning tasks is still untested. Furthermore, tackling reasoning tasks under the weak-to-strong setting currently lacks efficient methods to avoid blindly imitating the weak supervisor including its errors. In this paper, we introduce a progressive learning framework that enables the strong model to autonomously refine its training data, without requiring input from either a more advanced model or human-annotated data. This framework begins with supervised fine-tuning on a selective small but high-quality dataset, followed by preference optimization on contrastive samples identified by the strong model itself. Extensive experiments on the GSM8K and MATH datasets demonstrate that our method significantly enhances the reasoning capabilities of Llama2-70b using three separate weak models. This method is further validated in a forward-looking experimental setup, where Llama3-8b-instruct effectively supervises Llama3-70b on the highly challenging OlympicArena dataset. This work paves the way for a more scalable and sophisticated strategy to enhance AI reasoning powers. All relevant code and resources are available in urlhttps://github.com/GAIR-NLP/weak-to-strong-reasoning.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 46 of its 68 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

46 of 68 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Optimizing Language Model's Reasoning Abilities with Weak Supervision7 May
  2. 2024V-STaR: Training Verifiers for Self-Taught Reasoners9 Feb
  3. 2023LLMs cannot find reasoning errors, but can correct them given the error location14 Nov
  4. 2023Large Language Models Cannot Self-Correct Reasoning Yet3 Oct

Other references · 42

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  12. 2023LIMA: Less Is More for Alignmentcited by 1,080
30 more
  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  2. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  3. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  4. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  5. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  6. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  7. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  8. 2023Llemma: An Open Language Model For Mathematicscited by 385
  9. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  10. 2023The False Promise of Imitating Proprietary LLMscited by 254
  11. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  12. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  13. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  14. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  15. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  16. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  17. 2023Learning From Mistakes Makes LLM Better Reasonercited by 97
  18. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  19. 2024ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipelinecited by 47
  20. 2024Theoretical Analysis of Weak-to-Strong Generalizationcited by 46
  21. 2024The Unreasonable Effectiveness of Easy Training Data for Hard Taskscited by 40
  22. 2024Quantifying the Gain in Weak-to-Strong Generalizationcited by 40
  23. 2022Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutionscited by 38
  24. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysiscited by 34
  25. 2023AdaLomo: Low-memory Optimization with Adaptive Learning Ratecited by 30
  26. 2024Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Expertscited by 28
  27. 2023Plan, Verify and Switch: Integrated Reasoning with Diverse X-of-Thoughtscited by 27
  28. 2024Progress or Regress? Self-Improvement Reversal in Post-trainingcited by 23
  29. 2024Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learningcited by 17
  30. 2024A Moral Imperative: The Need for Continual Superalignment of Large Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 15; 2026: 9.

32 papers.

Sort:

By area:

  1. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 60
  2. 2024Can Language Models Learn to Skip Steps?cited by 59
  3. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 52
  4. 2025VisPlay: Self-Evolving Vision-Language Models from Imagescited by 42
  5. 2024Preference Optimization for Reasoning with Pseudo Feedbackcited by 38
  6. 2025AStar: Boosting Multimodal Reasoning with Automated Structured Thinkingcited by 25
  7. 2025The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gainscited by 25
  8. 2024MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimizationcited by 20
  9. 2026MM-Zero: Self-Evolving Multi-Model Vision Language Models From Zero Datacited by 16
  10. 2025Don't Pass@k: A Bayesian Framework for Large Language Model Evaluationcited by 15
  11. 2025Weak-to-Strong Generalization Even in Random Feature Networks, Provablycited by 13
  12. 2025HoneyBee: Data Recipes for Vision-Language Reasonerscited by 13
20 more
  1. 2025Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimensioncited by 10
  2. 2025Incentivizing Strong Reasoning from Weak Supervisioncited by 8
  3. 2024GFlowNet Fine-tuning for Diverse Correct Solutions in Mathematical Reasoning Taskscited by 7
  4. 2025On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspectivecited by 7
  5. 2026Weak-to-Strong On-Policy Distillationcited by 7
  6. 2026Weak-Driven Learning: How Weak Agents make Strong Agents Strongercited by 5
  7. 2026Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversightcited by 5
  8. 2025Self-Adaptive Cognitive Debiasing for Large Language Models in Decision-Makingcited by 4
  9. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  10. 2025Weak-to-Strong Generalization via Bregman Bias-Variance Decompositioncited by 3
  11. 2025Weak-to-Strong Generalization with Failure Trajectories: A Tree-based Approach to Elicit Optimal Policy in Strong Modelscited by 3
  12. 2025Mutual Learning for Hashing: Unlocking Strong Hash Functions from Weak Supervisioncited by 3
  13. 2026Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillationcited by 2
  14. 2024MATATA: Weakly Supervised End-to-End MAthematical Tool-Augmented Reasoning for Tabular Applicationscited by 1
  15. 2025Does Weak-to-strong Generalization Happen under Spurious Correlations?cited by 1
  16. 2026Weak-to-Strong Generalization is Nearly Inevitable (in Linear Models)cited by 1
  17. 2024Patience Is The Key to Large Language Model Reasoningcited by 0
  18. 2026Evaluating Risks in Weak-to-Strong Alignment: A Bias-Variance Perspectivecited by 0
  19. 2026On the Blessing of Pre-training in Weak-to-Strong Generalizationcited by 0
  20. 2026Allspark: Weak to Strong Transfer via Alternating Chain of Thoughtcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Synergistic Weak-Strong Collaboration by Aligning Preferences
  2. 2026Woodpecker Distillation: Weak Models Diagnose Reasoning Bugs in Strong Models
  3. 2024Weak-to-Strong Generalization beyond Accuracy: a Pilot Study in Safety, Toxicity, and Legal Reasoning
  4. 2024A transfer learning framework for weak-to-strong generalization
  5. 2024Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding
  6. 2025LightReasoner: Can Small Language Models Teach Large Language Models Reasoning?
  7. 2025Alice: Proactive Learning with Teacher's Demonstrations for Weak-to-Strong Generalization
  8. 2025Reasoning Transfer for an Extremely Low-Resource and Endangered Language: Bridging Languages Through Sample-Efficient Language Understanding