# Bayesian WeakS-to-Strong from Text Classification to Generation

arXiv: https://arxiv.org/abs/2406.03199 · HTML: https://mlsanity.com/paper/2406.03199/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 40 references that are in the map.
- First posted on arXiv 24 May 2024.

## Builds on (most cited first, 30 of 31)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://mlsanity.com/paper/1909.00277/) (2019)
- [Nash Learning from Human Feedback](https://mlsanity.com/paper/2312.00886/) (2023)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [On Releasing Annotator-Level Labels and Information in Datasets](https://mlsanity.com/paper/2110.05699/) (2021)
- [Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods](https://mlsanity.com/paper/2002.11955/) (2020)
- [Vision Superalignment: Weak-to-Strong Generalization for Vision Foundation Models](https://mlsanity.com/paper/2402.03749/) (2024)
- [Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Experts](https://mlsanity.com/paper/2402.15505/) (2024)
- [Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learning](https://mlsanity.com/paper/2402.00667/) (2024)
- [Estimating the Uncertainty in Emotion Attributes using Deep Evidential Regression](https://mlsanity.com/paper/2306.06760/) (2023)
- [Estimating the Uncertainty in Emotion Class Labels with Utterance-Specific Dirichlet Priors](https://mlsanity.com/paper/2203.04443/) (2022)
- [Knowledge-Aware Audio-Grounded Generative Slot Filling for Limited Annotated Data](https://mlsanity.com/paper/2307.01764/) (2023)

## Built on by (most cited first, 6 of 6)

- [Weak-to-Strong Generalization via Bregman Bias-Variance Decomposition](https://mlsanity.com/paper/2505.24313/) (2025)
- [Weak-to-Strong Generalization under Distribution Shifts](https://mlsanity.com/paper/2510.21332/) (2025)
- [How to Mitigate Overfitting in Weak-to-strong Generalization?](https://mlsanity.com/paper/2503.04249/) (2025)
- [On the Blessing of Pre-training in Weak-to-Strong Generalization](https://mlsanity.com/paper/2605.05710/) (2026)
- [Denoising Mutual Knowledge Distillation in Bi-Directional Multiple Instance Learning](https://mlsanity.com/paper/2505.12074/) (2025)
- [Debate-to-Skill: Capability-Bound Process Supervision for Industrial Query-to-Agent Annotation](https://mlsanity.com/paper/2609.11176/) (2026)

## Abstract (arXiv)

Advances in large language models raise the question of how alignment techniques will adapt as models become increasingly complex and humans will only be able to supervise them weakly. Weak-to-Strong mimics such a scenario where weak model supervision attempts to harness the full capabilities of a much stronger model. This work extends Weak-to-Strong to WeakS-to-Strong by exploring an ensemble of weak models which simulate the variability in human opinions. Confidence scores are estimated using a Bayesian approach to guide the WeakS-to-Strong generalization. Furthermore, we extend the application of WeakS-to-Strong from text classification tasks to text generation tasks where more advanced strategies are investigated for supervision. Moreover, direct preference optimization is applied to advance the student model's preference learning, beyond the basic learning framework of teacher forcing. Results demonstrate the effectiveness of the proposed approach for the reliability of a strong student model, showing potential for superalignment.