# Guided Self-Evolving LLMs with Minimal Human Supervision

arXiv: https://arxiv.org/abs/2512.02472 · HTML: https://mlsanity.com/paper/2512.02472/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 50 references that are in the map.
- First posted on arXiv 2 Dec 2025.

## Builds on (most cited first, 30 of 43)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [Absolute Zero: Reinforced Self-play Reasoning with Zero Data](https://mlsanity.com/paper/2505.03335/) (2025)
- [Reasoning with Exploration: An Entropy Perspective](https://mlsanity.com/paper/2506.14758/) (2025)
- [TTRL: Test-Time Reinforcement Learning](https://mlsanity.com/paper/2504.16084/) (2025)
- [A Survey of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2312.14925/) (2023)
- [R-Zero: Self-Evolving Reasoning LLM from Zero Data](https://mlsanity.com/paper/2508.05004/) (2025)
- [Learning to Reason without External Rewards](https://mlsanity.com/paper/2505.19590/) (2025)
- [The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning](https://mlsanity.com/paper/2505.15134/) (2025)
- [Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains](https://mlsanity.com/paper/2503.23829/) (2025)
- [Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play](https://mlsanity.com/paper/1703.05407/) (2017)
- [General-Reasoner: Advancing LLM Reasoning Across All Domains](https://mlsanity.com/paper/2505.14652/) (2025)
- [BIG-Bench Extra Hard](https://mlsanity.com/paper/2502.19187/) (2025)
- [Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning](https://mlsanity.com/paper/2504.03380/) (2025)
- [Maximizing Confidence Alone Improves Reasoning](https://mlsanity.com/paper/2505.22660/) (2025)
- [Efficient Reinforcement Finetuning via Adaptive Curriculum Learning](https://mlsanity.com/paper/2504.05520/) (2025)
- [A Survey on Self-Evolution of Large Language Models](https://mlsanity.com/paper/2404.14387/) (2024)
- [SPICE: Self-Play In Corpus Environments Improves Reasoning](https://mlsanity.com/paper/2510.24684/) (2025)

## Built on by (most cited first, 30 of 35)

- [UniCorn: Towards Self-Improving Unified Multimodal Models through Self-Generated Supervision](https://mlsanity.com/paper/2601.03193/) (2026)
- [Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verification](https://mlsanity.com/paper/2601.15808/) (2026)
- [From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms](https://mlsanity.com/paper/2605.06716/) (2026)
- [TTCS: Test-Time Curriculum Synthesis for Self-Evolving](https://mlsanity.com/paper/2601.22628/) (2026)
- [G-Zero: Self-Play for Open-Ended Generation from Zero Data](https://mlsanity.com/paper/2605.09959/) (2026)
- [Verified Critical Step Optimization for LLM Agents](https://mlsanity.com/paper/2602.03412/) (2026)
- [How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization](https://mlsanity.com/paper/2602.19208/) (2026)
- [Self-Evolving World Models for LLM Agent Planning](https://mlsanity.com/paper/2606.30639/) (2026)
- [Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning](https://mlsanity.com/paper/2601.18984/) (2026)
- [R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training](https://mlsanity.com/paper/2602.13103/) (2026)
- [Scaling Self-Play with Self-Guidance](https://mlsanity.com/paper/2604.20209/) (2026)
- [Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills](https://mlsanity.com/paper/2607.22529/) (2026)
- [RelayLLM: Efficient Reasoning via Collaborative Decoding](https://mlsanity.com/paper/2601.05167/) (2026)
- [Darwinian Memory: A Training-Free Self-Regulating Memory System for GUI Agent Evolution](https://mlsanity.com/paper/2601.22528/) (2026)
- [Self-Play Meets Skill Evolution: Self-Evolving Search Agents that Pose, Solve, and Remember](https://mlsanity.com/paper/2607.29468/) (2026)
- [DARC: Decoupled Asymmetric Reasoning Curriculum for LLM Evolution](https://mlsanity.com/paper/2601.13761/) (2026)
- [CoCo: Code as CoT for Text-to-Image Preview and Rare Concept Generation](https://mlsanity.com/paper/2603.08652/) (2026)
- [Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding](https://mlsanity.com/paper/2604.21268/) (2026)
- [Adaptive Time Series Reasoning via Segment Selection](https://mlsanity.com/paper/2602.18645/) (2026)
- [GASP: Guided Asymmetric Self-Play For Coding LLMs](https://mlsanity.com/paper/2603.15957/) (2026)
- [Preventing Curriculum Collapse in Self-Evolving Reasoning Systems](https://mlsanity.com/paper/2603.13309/) (2026)
- [MAGMA-GEN: Validated Recovery Supervision from Ambiguous Failures via Counterfactual Re-Execution](https://mlsanity.com/paper/2609.20056/) (2026)
- [DuCCAE: A Hybrid Engine for Immersive Conversation via Collaboration, Augmentation, and Evolution](https://mlsanity.com/paper/2603.19248/) (2026)
- [Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution](https://mlsanity.com/paper/2604.03472/) (2026)
- [Seirenes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning](https://mlsanity.com/paper/2605.11636/) (2026)
- [RISE: Reliable Improvement in Self-Evolving Vision-Language Models](https://mlsanity.com/paper/2605.20914/) (2026)
- [ANCHOR: An External LLM-Driven Supervisory Module Facilitating Healthy Evolution in Self-Evolving Systems](https://mlsanity.com/paper/2606.06114/) (2026)
- [Anchored Self-Play for Code Repair](https://mlsanity.com/paper/2607.03523/) (2026)
- [Learning What to Practice: Diagnosis-Guided Self-Evolution for Language Models](https://mlsanity.com/paper/2609.00768/) (2026)
- [Sci-CoE: Co-evolving Scientific Reasoning LLMs via Geometric Consensus with Sparse Supervision](https://mlsanity.com/paper/2602.12164/) (2026)

## Abstract (arXiv)

AI self-evolution has long been envisioned as a path toward superintelligence, where models autonomously acquire, refine, and internalize knowledge from their own learning experiences. Yet in practice, unguided self-evolving systems often plateau quickly or even degrade as training progresses. These failures arise from issues such as concept drift, diversity collapse, and mis-evolution, as models reinforce their own biases and converge toward low-entropy behaviors. To enable models to self-evolve in a stable and controllable manner while minimizing reliance on human supervision, we introduce R-Few, a guided Self-Play Challenger-Solver framework that incorporates lightweight human oversight through in-context grounding and mixed training. At each iteration, the Challenger samples a small set of human-labeled examples to guide synthetic question generation, while the Solver jointly trains on human and synthetic examples under an online, difficulty-based curriculum. Across math and general reasoning benchmarks, R-Few achieves consistent and iterative improvements. For example, Qwen3-8B-Base improves by +3.0 points over R-Zero on math tasks and achieves performance on par with General-Reasoner, despite the latter being trained on 20 times more human data. Ablation studies confirm the complementary contributions of grounded challenger training and curriculum-based solver training, and further analysis shows that R-Few mitigates drift, yielding more stable and controllable co-evolutionary dynamics.