mlsanityData to 14 Oct 2026

Weak-to-Strong Generalization with Failure Trajectories: A Tree-based Approach to Elicit Optimal Policy in Strong Models

arXiv 2507.18858 ↗First posted 25 Jul 2025cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Weak-to-Strong generalization (W2SG) is a new trend to elicit the full capabilities of a strong model with supervision from a weak model. While existing W2SG studies focus on simple tasks like binary classification, we extend this paradigm to complex interactive decision-making environments. Specifically, we fine-tune a strong model with trajectories of intermediate actions generated by a weak model. Motivated by the human learning process, we propose to generalize not only success knowledge but also failure experience so that the strong model can learn from failed trajectories accumulated by weak models. To effectively and efficiently elicit the potential of strong agents, we further construct ``trajectory trees," a hierarchical representation that organizes weak model-generated action trajectories, coupled with Monte Carlo Tree Search (MCTS) to optimize the strong model. Through theoretical analysis, we provide formal guarantees for the effectiveness of our method in improving W2SG performance. Our empirical evaluations demonstrate substantial improvements in reasoning and decision-making capabilities across diverse task domains, validating the scalability and robustness of our proposed framework.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 42 of its 54 references that are in the map. First posted on arXiv 25 Jul 2025.

Builds on

42 of 54 references.

References · 42

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  10. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  11. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  12. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
30 more
  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  3. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  4. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  5. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 392
  6. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  7. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 327
  8. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
  9. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 197
  10. 2024Direct Preference Optimization with an Offsetcited by 122
  11. 2024Faithful Logical Reasoning via Symbolic Chain-of-Thoughtcited by 100
  12. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 97
  13. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 96
  14. 2020Monte-Carlo Tree Search as Regularized Policy Optimizationcited by 61
  15. 2023Beyond Chain-of-Thought, Effective Graph-of-Thought Reasoning in Language Modelscited by 52
  16. 2024Theoretical Analysis of Weak-to-Strong Generalizationcited by 48
  17. 2024Quantifying the Gain in Weak-to-Strong Generalizationcited by 41
  18. 2024Weak-to-Strong Reasoningcited by 33
  19. 2023Agent Instructs Large Language Models to be General Zero-Shot Reasonerscited by 32
  20. 2023Towards a statistical theory of data selection under weak supervisioncited by 27
  21. 2024Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalizationcited by 23
  22. 2024MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimizationcited by 22
  23. 2024Weak-to-Strong Generalization Through the Data-Centric Lenscited by 21
  24. 2025QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Searchcited by 20
  25. 2024Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learningcited by 18
  26. 2023Revisit Weakly-Supervised Audio-Visual Video Parsing from the Language Perspectivecited by 14
  27. 2024A General Framework for Learning from Weak Supervisioncited by 7
  28. 2024Generalizing Trust: Weak-to-Strong Trustworthiness in Language Modelscited by 6
  29. 2024Weak-to-Strong Generalization beyond Accuracy: a Pilot Study in Safety, Toxicity, and Legal Reasoningcited by 4
  30. 2025How to Mitigate Overfitting in Weak-to-strong Generalization?cited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Weak-to-Strong Generalization is Nearly Inevitable (in Linear Models)cited by 2
  2. 2026From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Mergingcited by 1
  3. 2026Trust Functions: Near-Lossless Weak-to-Strong Generalization by Learning When to Trust the Weak Teachercited by 0