mlsanityData to 7 Oct 2026

Group-Marginalized Self-Rewarding RL Drives Zero-Label Self-Evolving

arXiv 2609.36750 ↗First posted 29 Sep 2026cs.LG cs.AI cs.CLcited by 0

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

Self-rewarding reinforcement learning (RL) enables large language models (LLMs) to self-evolve without human labels. Existing ensemble-based methods construct reward references from rollout groups and assign rewards accordingly. However, a response's reward representation also depends on its randomly sampled group context, i.e., the other responses in its group. Using only one group-context realization may miss desired reward signals and provide unreliable guidance for policy optimization. To address this issue, we propose Group-Marginalized Advantage Estimation (GMAE), which aggregates reward realizations across possible contexts into a response-level distribution and estimates expected advantages. Experiments across eight benchmarks and four base models demonstrate strong performance and cross-domain generalization. GMAE also exhibits stable learning, low extra cost, and good applicability across training datasets and RL backbones.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 47 of its 48 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

47 of 48 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge28 Jul
  2. 2024Self-Rewarding Language Models18 Jan
  3. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models2 Jan
  4. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug

Other references · 43

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
31 more
  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  6. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  7. 2025Group Sequence Policy Optimizationcited by 788
  8. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  9. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  10. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  11. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  12. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 233
  13. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  14. 2025Learning to Reason without External Rewardscited by 199
  15. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  16. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  17. 2024Self-Discover: Large Language Models Self-Compose Reasoning Structurescited by 108
  18. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  19. 2025Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn'tcited by 72
  20. 2024Self-Taught Evaluatorscited by 70
  21. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 69
  22. 2025Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Modelscited by 61
  23. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoningcited by 43
  24. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoningcited by 35
  25. 2026How Far Can Unsupervised RLVR Scale LLM Training?cited by 26
  26. 2025Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Modelscited by 20
  27. 2025RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalizationcited by 13
  28. 2025Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learningcited by 11
  29. 2025Self-Harmony: Learning to Harmonize Self-Supervision and Self-Play in Test-Time Reinforcement Learningcited by 9
  30. 2025You Need Reasoning to Learn Reasoning: The Limitations of Label-Free RL in Weak Base Modelscited by 2
  31. 2026On the Overscaling Curse of Parallel Thinking: System Efficacy Contradicts Sample Efficiencycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning
  2. 2026Your Group-Relative Advantage Is Biased
  3. 2026Rethinking Groups in Critic-Free RLVR
  4. 2026Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
  5. 2026Save Your Saturated Data: Learning Beyond Reward Saturation in Group-Based RL
  6. 2026Pairwise Preference Reward and Group-Based Diversity Enhancement for Superior Open-Ended Generation
  7. 2026On the Hidden Objective Biases of Group-based Reinforcement Learning
  8. 2025Group-in-Group Policy Optimization for LLM Agent Training