mlsanityData to 7 Oct 2026

Weak-to-Strong Generalization Even in Random Feature Networks, Provably

arXiv 2503.02877 ↗First posted 4 Mar 2025cs.LG stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

Weak-to-Strong Generalization (Burns et al., 2024) is the phenomenon whereby a strong student, say GPT-4, learns a task from a weak teacher, say GPT-2, and ends up significantly outperforming the teacher. We show that this phenomenon does not require a strong learner like GPT-4. We consider student and teacher that are random feature models, described by two-layer networks with a random and fixed bottom layer and a trained top layer. A "weak" teacher, with a small number of units (i.e. random features), is trained on the population, and a "strong" student, with a much larger number of units (i.e. random features), is trained only on labels generated by the weak teacher. We demonstrate, prove, and understand how the student can outperform the teacher, even though trained only on data labeled by the teacher. We also explain how such weak-to-strong generalization is enabled by early stopping. Importantly, we also show the quantitative limits of weak-to-strong generalization in this model.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

22 of 30 references.

References · 22

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  3. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  4. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  5. 2024Aligner: Efficient Alignment by Learning to Correctcited by 96
  6. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
  7. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 66
  8. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 60
  9. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 54
  10. 2024Theoretical Analysis of Weak-to-Strong Generalizationcited by 46
  11. 2024Quantifying the Gain in Weak-to-Strong Generalizationcited by 40
  12. 2024Weak-to-Strong Reasoningcited by 32
10 more
  1. 2021The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networkscited by 25
  2. 2022Learning sparse features can lead to overfitting in neural networkscited by 23
  3. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  4. 2024A non-asymptotic theory of Kernel Ridge Regression: deterministic equivalents, test error, and GCV estimatorcited by 22
  5. 2024High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Lawscited by 21
  6. 2024Weak-to-Strong Generalization Through the Data-Centric Lenscited by 21
  7. 2024Your Weak LLM is Secretly a Strong Teacher for Alignmentcited by 20
  8. 2024Provable Weak-to-Strong Generalization via Benign Overfittingcited by 16
  9. 2024A transfer learning framework for weak-to-strong generalizationcited by 14
  10. 2025Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Losscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 8.

13 papers.

Sort:

By area:

  1. 2025On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspectivecited by 7
  2. 2025On Learning Verifiers and Implications to Chain-of-Thought Reasoningcited by 6
  3. 2025Weak-to-Strong Generalization via Bregman Bias-Variance Decompositioncited by 3
  4. 2025Can Small Training Runs Reliably Guide Data Curation? Rethinking Proxy-Model Practicecited by 3
  5. 2026Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillationcited by 2
  6. 2025Does Weak-to-strong Generalization Happen under Spurious Correlations?cited by 1
  7. 2026Weak-to-Strong Generalization is Nearly Inevitable (in Linear Models)cited by 1
  8. 2026The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledgecited by 1
  9. 2026Why Self-Training Helps and Hurts: Denoising vs. Signal Forgettingcited by 0
  10. 2026On the Blessing of Pre-training in Weak-to-Strong Generalizationcited by 0
  11. 2026Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descentcited by 0
  12. 2026Trust Functions: Near-Lossless Weak-to-Strong Generalization by Learning When to Trust the Weak Teachercited by 0
1 more
  1. 2026What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regressioncited by 0