# Reinforcement Learning from Meta-Evaluation: Aligning Language Models Without Ground-Truth Labels

arXiv: https://arxiv.org/abs/2601.21268 · HTML: https://mlsanity.com/paper/2601.21268/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2026.

## Built on by (most cited first, 3 of 3)

- [Self-evolving Embodied AI](https://mlsanity.com/paper/2602.04411/) (2026)
- [RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges](https://mlsanity.com/paper/2608.24231/) (2026)
- [Unsupervised Post-Training of Foundation Models: A Survey](https://mlsanity.com/paper/2608.24982/) (2026)

## Abstract (arXiv)

Most reinforcement learning (RL) methods for training large language models (LLMs) require ground-truth labels or task-specific verifiers, limiting scalability when correctness is ambiguous or expensive to obtain. We introduce Reinforcement Learning from Meta-Evaluation (RLME), which optimizes a generator using reward derived from an evaluator's answers to natural-language meta-questions (e.g., "Is the answer correct?" or "Is the reasoning logically consistent?"). RLME treats the evaluator's probability of a positive judgment as a reward and updates the generator via group-relative policy optimization, enabling learning without labels. Across a suite of experiments, we show that RLME achieves accuracy and sample efficiency comparable to label-based training, enables controllable trade-offs among multiple objectives, steers models toward reliable reasoning patterns rather than post-hoc rationalization, and generalizes to open-domain settings where ground-truth labels are unavailable, broadening the domains in which LLMs may be trained with RL.