# On Weak-to-Strong Generalization and f-Divergence

arXiv: https://arxiv.org/abs/2506.03109 · HTML: https://mlsanity.com/paper/2506.03109/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2025.

## Built on by (most cited first, 4 of 4)

- [Weak-to-Strong On-Policy Distillation](https://mlsanity.com/paper/2607.26246/) (2026)
- [Weak-to-Strong Generalization is Nearly Inevitable (in Linear Models)](https://mlsanity.com/paper/2605.05742/) (2026)
- [Empirical Risk Minimization with f-Divergence Regularization](https://mlsanity.com/paper/2601.13191/) (2026)
- [AdaStop: Cost-Aware Early Stopping for DNN Test Selection](https://mlsanity.com/paper/2607.05461/) (2026)

## Abstract (arXiv)

Weak-to-strong generalization (W2SG) has emerged as a promising paradigm for stimulating the capabilities of strong pre-trained models by leveraging supervision from weaker supervisors. To improve the performance of the strong model, existing methods often require additional weak models or complex procedures, leading to substantial computational and memory overhead. Motivated by the effectiveness of f-divergence loss in various machine learning domains, we introduce f-divergence as an information-theoretic loss function framework in W2SG. Our theoretical analysis reveals fundamental limitations and equivalence of different f-divergence losses in W2SG, supported by sample complexity bounds and information-theoretic insights. We empirically demonstrate that f-divergence loss, which generalizes widely-used metrics like KL divergence, effectively improves generalization and noise tolerance of the strong model in practice.