mlsanityData to 14 Oct 2026

G-Zero: Self-Play for Open-Ended Generation from Zero Data

arXiv 2605.09959 ↗First posted 11 May 2026cs.LG cs.AI cs.CLcited by 17

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

also in 1 more
Abstract (arXiv)

Self-evolving LLMs excel in verifiable domains but struggle in open-ended tasks, where reliance on proxy LLM judges introduces capability bottlenecks and reward hacking. To overcome this, we introduce G-Zero, a verifier-free, co-evolutionary framework for autonomous self-improvement. Our core innovation is Hint-δ, an intrinsic reward that quantifies the predictive shift between a Generator model's unassisted response and its response conditioned on a self-generated hint. Using this signal, a Proposer model is trained via GRPO to continuously target the Generator's blind spots by synthesizing challenging queries and informative hints. The Generator is concurrently optimized via DPO to internalize these hint-guided improvements. Theoretically, we prove a best-iterate suboptimality guarantee for an idealized standard-DPO version of G-Zero, provided that the Proposer induces sufficient exploration coverage and the data filteration keeps pseudo-label score noise low. By deriving supervision entirely from internal distributional dynamics, G-Zero bypasses the capability ceilings of external judges, providing a scalable, robust pathway for continuous LLM self-evolution across unverifiable domains.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 39 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

33 of 39 references.

References · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  6. 2024A Survey on LLM-as-a-Judgecited by 1,423
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  8. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  10. 2022Large Language Models Can Self-Improvecited by 664
  11. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  12. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
21 more
  1. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  2. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 221
  3. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 205
  4. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  5. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 150
  6. 2024A Survey on Self-Evolution of Large Language Modelscited by 91
  7. 2025RLPR: Extrapolating RLVR to General Domains without Verifierscited by 85
  8. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 83
  9. 2025Reinforcing General Reasoning without Verifierscited by 77
  10. 2025Self-Challenging Language Model Agentscited by 59
  11. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 51
  12. 2018FIRST: A Framework for Optimizing Information Quality in Mobile Crowdsensing Systemscited by 50
  13. 2025VisPlay: Self-Evolving Vision-Language Models from Imagescited by 47
  14. 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoningcited by 42
  15. 2025Learning to Solve and Verify: A Self-Play Framework for Code and Test Generationcited by 35
  16. 2025Guided Self-Evolving LLMs with Minimal Human Supervisioncited by 35
  17. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  18. 2026MM-Zero: Self-Evolving Multi-Model Vision Language Models From Zero Datacited by 21
  19. 2026Likelihood-Based Reward Designs for General LLM Reasoningcited by 7
  20. 2026Efficient Paths and Dense Rewards: Probabilistic Flow Reasoning for Large Language Modelscited by 2
  21. 2026Reinforcement Learning with Conditional Expectation Rewardcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 17.

17 papers.

Sort:

By area:

  1. 2026Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loopscited by 35
  2. 2026Dream-RSI: Recursive Self-Improvement through Evolving Worldscited by 18
  3. 2026EnvHarness: Awakening Static Worlds for Agent Learningcited by 13
  4. 2026RRSI: Regularized Recursive Self-Improvement of Agent Harnessescited by 13
  5. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 12
  6. 2026More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judgescited by 10
  7. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  8. 2026SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learningcited by 2
  9. 2026It Takes Two to Match: Co-Evolving Generative Retriever with Reinforcement Learningcited by 2
  10. 2026Negative Self-Distillation: Learning to Reason by Avoiding Flawscited by 2
  11. 2026J-Zero: Unified Challenger--Solver--Judge Self-Evolution from Zero Datacited by 1
  12. 2026Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Designcited by 1
5 more
  1. 2026Unsupervised Post-Training of Foundation Models: A Surveycited by 0
  2. 2026STRETCH the Boundaries: A Unified Self-Taught Framework for Progressive LLM Evolutioncited by 0
  3. 2026When Updating Stops Being Learning: Rethinking LLM Self-Evolution via learnable information gaincited by 0
  4. 2026Questioning the Questions: Sustaining Self-Evolution in Reasoning Modelscited by 0
  5. 2026From Existing Methods to New Research Ideas: Two Perspectives on Recursive Self-Improvementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Compute Allocation for Self-Evolving LLMs: From Depth-Breadth to Multi-Armed Bandits
  2. 2026Teach to Learn: Hint Annealing for Self-improving LLM Reasoning
  3. 2025EvoSyn: Generalizable Evolutionary Data Synthesis for Verifiable Learning
  4. 2026The Red Queen G"odel Machine: Co-Evolving Agents and Their Evaluators
  5. 2024Can Large Language Models Invent Algorithms to Improve Themselves?: Algorithm Discovery for Recursive Self-Improvement through Reinforcement Learning
  6. 2026Beyond the Capability Boundary: Zeroth-Order Optimization for Self-Evolving LLM Agents
  7. 2026Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning
  8. 2026Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gain