# Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Experts

arXiv: https://arxiv.org/abs/2402.15505 · HTML: https://mlsanity.com/paper/2402.15505/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts.
- Builds on 31 of its 46 references that are in the map.
- First posted on arXiv 23 Feb 2024.

## Placed in

- Mixture-of-experts / [Vision mixture-of-experts](https://mlsanity.com/moe/vision-mixture-of-experts/)

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [Ensemble deep learning: A review](https://mlsanity.com/paper/2104.02395/) (2021)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019)
- [Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise](https://mlsanity.com/paper/1802.05300/) (2018)
- [Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels](https://mlsanity.com/paper/1905.05040/) (2019)
- [A Review of Sparse Expert Models in Deep Learning](https://mlsanity.com/paper/2209.01667/) (2022)
- [Supervising strong learners by amplifying weak experts](https://mlsanity.com/paper/1810.08575/) (2018)
- [Evaluating Superhuman Models with Consistency Checks](https://mlsanity.com/paper/2306.09983/) (2023)
- [Embedded Agency](https://mlsanity.com/paper/1902.09469/) (2019)

## Built on by (most cited first, 28 of 28)

- [What is the Role of Small Models in the LLM Era: A Survey](https://mlsanity.com/paper/2409.06857/) (2024)
- [Weak-to-Strong Reasoning](https://mlsanity.com/paper/2407.13647/) (2024)
- [MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization](https://mlsanity.com/paper/2410.07672/) (2024)
- [Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executors](https://mlsanity.com/paper/2504.04785/) (2025)
- [Your Weak LLM is Secretly a Strong Teacher for Alignment](https://mlsanity.com/paper/2409.08813/) (2024)
- [Provable Weak-to-Strong Generalization via Benign Overfitting](https://mlsanity.com/paper/2410.04638/) (2024)
- [Improving Weak-to-Strong Generalization with Reliability-Aware Alignment](https://mlsanity.com/paper/2406.19032/) (2024)
- [The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignment](https://mlsanity.com/paper/2412.16468/) (2024)
- [Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimension](https://mlsanity.com/paper/2502.05075/) (2025)
- [ALaRM: Align Language Models via Hierarchical Rewards Modeling](https://mlsanity.com/paper/2403.06754/) (2024)
- [Hummer: Towards Limited Competitive Preference Dataset](https://mlsanity.com/paper/2405.11647/) (2024)
- [Efficient Model-agnostic Alignment via Bayesian Persuasion](https://mlsanity.com/paper/2405.18718/) (2024)
- [Debate Helps Weak-to-Strong Generalization](https://mlsanity.com/paper/2501.13124/) (2025)
- [On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective](https://mlsanity.com/paper/2505.18346/) (2025)
- [Incentivizing Strong Reasoning from Weak Supervision](https://mlsanity.com/paper/2505.20072/) (2025)
- [Bayesian WeakS-to-Strong from Text Classification to Generation](https://mlsanity.com/paper/2406.03199/) (2024)
- [Generalizable Video Quality Assessment via Weak-to-Strong Learning](https://mlsanity.com/paper/2505.03631/) (2025)
- [EnsemW2S: Enhancing Weak-to-Strong Generalization with Large Language Model Ensembles](https://mlsanity.com/paper/2505.21959/) (2025)
- [Weak-to-Strong Generalization via Bregman Bias-Variance Decomposition](https://mlsanity.com/paper/2505.24313/) (2025)
- [Weak-to-Strong Generalization under Distribution Shifts](https://mlsanity.com/paper/2510.21332/) (2025)
- [How to Mitigate Overfitting in Weak-to-strong Generalization?](https://mlsanity.com/paper/2503.04249/) (2025)
- [The Capabilities and Limitations of Weak-to-Strong Generalization: Generalization and Calibration](https://mlsanity.com/paper/2502.01458/) (2025)
- [Collaborative Learning of On-Device Small Model and Cloud-Based Large Model: Advances and Future Directions](https://mlsanity.com/paper/2504.15300/) (2025)
- [Does Weak-to-strong Generalization Happen under Spurious Correlations?](https://mlsanity.com/paper/2509.24005/) (2025)
- [EnsemW2S: Enhancing Weak-to-Strong Generalization with Large Language Model Ensembles](https://mlsanity.com/paper/2410.04571/) (2024)
- [Research Superalignment Should Advance Now with Alternating Competence and Conformity Optimization](https://mlsanity.com/paper/2503.07660/) (2025)
- [On the Blessing of Pre-training in Weak-to-Strong Generalization](https://mlsanity.com/paper/2605.05710/) (2026)
- [Trust Functions: Near-Lossless Weak-to-Strong Generalization by Learning When to Trust the Weak Teacher](https://mlsanity.com/paper/2606.01000/) (2026)

## Abstract (arXiv)

Steering the behavior of a strong model pre-trained on internet-scale data can be difficult due to the scarcity of competent supervisors. Recent studies reveal that, despite supervisory noises, a strong student model may surpass its weak teacher when fine-tuned on specific objectives. Yet, the effectiveness of such weak-to-strong generalization remains limited, especially in the presence of large capability gaps. In this paper, we propose to address this challenge by harnessing a diverse set of specialized teachers, instead of a single generalist one, that collectively supervises the strong student. Our approach resembles the classical hierarchical mixture of experts, with two components tailored for co-supervision: (i) we progressively alternate student training and teacher assignment, leveraging the growth of the strong student to identify plausible supervisions; (ii) we conservatively enforce teacher-student and local-global consistency, leveraging their dependencies to reject potential annotation noises. We validate the proposed method through visual recognition tasks on the OpenAI weak-to-strong benchmark and additional multi-domain datasets. Our code is available at urlhttps://github.com/yuejiangliu/csl.