# Why Self-Rewarding Works: Theoretical Guarantees for Iterative Alignment of Language Models

arXiv: https://arxiv.org/abs/2601.22513 · HTML: https://mlsanity.com/paper/2601.22513/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 24 of its 66 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)
- Synthetic data and self-improvement for LLMs / [Self-rewarding alignment and self-training](https://mlsanity.com/synth/self-rewarding-alignment-and-self-training/)

## Builds on (most cited first, 24 of 24)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring](https://mlsanity.com/paper/1911.09785/) (2019)
- [Learning to Reason without External Rewards](https://mlsanity.com/paper/2505.19590/) (2025)
- [Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge](https://mlsanity.com/paper/2407.19594/) (2024)
- [The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning](https://mlsanity.com/paper/2505.15134/) (2025)
- [Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization](https://mlsanity.com/paper/2504.05812/) (2025)
- [What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization](https://mlsanity.com/paper/2305.19420/) (2023)
- [Self-Rewarding Vision-Language Model via Reasoning Decomposition](https://mlsanity.com/paper/2508.19652/) (2025)
- [Model Collapse Demystified: The Case of Regression](https://mlsanity.com/paper/2402.07712/) (2024)
- [Self-rewarding correction for mathematical reasoning](https://mlsanity.com/paper/2502.19613/) (2025)
- [Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding](https://mlsanity.com/paper/2411.04282/) (2024)
- [Rate of Model Collapse in Recursive Training](https://mlsanity.com/paper/2412.17646/) (2024)
- [Heat Death of Generative Models in Closed-Loop Learning](https://mlsanity.com/paper/2404.02325/) (2024)
- [Large Language Models Encode Semantics and Alignment in Linearly Separable Representations](https://mlsanity.com/paper/2507.09709/) (2025)
- [Towards a theory of model distillation](https://mlsanity.com/paper/2403.09053/) (2024)
- [Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models](https://mlsanity.com/paper/2502.08922/) (2025)
- [PACR: Progressively Ascending Confidence Reward for LLM Reasoning](https://mlsanity.com/paper/2510.22255/) (2025)
- [Confidence as a Reward: Transforming LLMs into Reward Models](https://mlsanity.com/paper/2510.13501/) (2025)
- [Semi-supervised Batch Learning From Logged Data](https://mlsanity.com/paper/2209.07148/) (2022)
- [The Best of Both Worlds: Reinforcement Learning with Logarithmic Regret and Policy Switches](https://mlsanity.com/paper/2203.01491/) (2022)
- [Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only](https://mlsanity.com/paper/2510.21090/) (2025)

## Built on by (most cited first, 1 of 1)

- [Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops](https://mlsanity.com/paper/2607.07663/) (2026)

## Abstract (arXiv)

Self-Rewarding Language Models (SRLMs) achieve notable success in iteratively improving alignment without external feedback. Yet, despite their striking empirical progress, the core mechanisms driving their capabilities remain unelucidated, leaving a critical gap in theoretical understanding. This paper provides the first rigorous theoretical guarantees for SRLMs. We first establish a lower bound that characterizes the fundamental limits of a single update step, revealing a critical dependence on the quality of the initial model. We then derive finite-sample error bounds for the full iterative paradigm, showing that performance improves at a rate of widetildeOleft(1/√nright) with sample size n. Crucially, our analysis reveals that the dependence on the initial model decays exponentially with the number of iterations T. This provides a formal explanation for why self-rewarding succeeds: it robustly overcomes poor initialization by steering the dynamics toward internal stability and consistency. Finally, we instantiate our theoretical framework for the linear softmax model class, yielding tailored guarantees that connect our high-level insights to practical model architectures.