mlsanityData to 7 Oct 2026

Bayesian WeakS-to-Strong from Text Classification to Generation

arXiv 2406.03199 ↗First posted 24 May 2024cs.CL cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Advances in large language models raise the question of how alignment techniques will adapt as models become increasingly complex and humans will only be able to supervise them weakly. Weak-to-Strong mimics such a scenario where weak model supervision attempts to harness the full capabilities of a much stronger model. This work extends Weak-to-Strong to WeakS-to-Strong by exploring an ensemble of weak models which simulate the variability in human opinions. Confidence scores are estimated using a Bayesian approach to guide the WeakS-to-Strong generalization. Furthermore, we extend the application of WeakS-to-Strong from text classification tasks to text generation tasks where more advanced strategies are investigated for supervision. Moreover, direct preference optimization is applied to advance the student model's preference learning, beyond the basic learning framework of teacher forcing. Results demonstrate the effectiveness of the proposed approach for the reliability of a strong student model, showing potential for superalignment.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 31 of its 40 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

31 of 40 references.

References · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
19 more
  1. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  2. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  3. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 728
  4. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  5. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  6. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  7. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
  8. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  9. 2023Nash Learning from Human Feedbackcited by 220
  10. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
  11. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 123
  12. 2020Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methodscited by 98
  13. 2024Vision Superalignment: Weak-to-Strong Generalization for Vision Foundation Modelscited by 28
  14. 2024Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Expertscited by 28
  15. 2024Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learningcited by 17
  16. 2023Estimating the Uncertainty in Emotion Attributes using Deep Evidential Regressioncited by 11
  17. 2022Estimating the Uncertainty in Emotion Class Labels with Utterance-Specific Dirichlet Priorscited by 8
  18. 2023Knowledge-Aware Audio-Grounded Generative Slot Filling for Limited Annotated Datacited by 3
  19. 2023Speech-based Slot Filling using Large Language Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Weak-to-Strong Generalization under Distribution Shiftscited by 4
  2. 2025How to Mitigate Overfitting in Weak-to-strong Generalization?cited by 3
  3. 2025Weak-to-Strong Generalization via Bregman Bias-Variance Decompositioncited by 3
  4. 2025Denoising Mutual Knowledge Distillation in Bi-Directional Multiple Instance Learningcited by 0
  5. 2026On the Blessing of Pre-training in Weak-to-Strong Generalizationcited by 0
  6. 2026Debate-to-Skill: Capability-Bound Process Supervision for Industrial Query-to-Agent Annotationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model
  2. 2024Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models
  3. 2024Weak-to-Strong Generalization beyond Accuracy: a Pilot Study in Safety, Toxicity, and Legal Reasoning
  4. 2024Improving Weak-to-Strong Generalization with Reliability-Aware Alignment
  5. 2026When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger
  6. 2024A transfer learning framework for weak-to-strong generalization
  7. 2025Contrastive Weak-to-strong Generalization
  8. 2024Weak-to-Strong Reasoning